Edition：1
Investigation report for Deviation #55540
Edition: 1
The Batch Cleaning (APSLY50H-V1-CL-25001) of V Plant Line 1 after the execution of APS in Freeze Dryer (FZ-V209) has triggered an abnormal alarm on the HMI and cannot be reset
Written by: | |
Reviewed by: | |
Reviewed by: | |
Approved by: |
Edition：1
Table of Contents
Edition：1
Short Description
The production line V1 of plant V was cleaning after APS batch(APSLY50H-V1-CL-25001) in the freeze dryer (FZ-V209). During CIP , the equipment HMI popped up an abnormal alarm and could not be restored
Description of Deviation
The V1 production line at the V plant performed APS filling batch (Batch number: APSLY50H-V1-25001) on 02/15/2025. After the batch, the line was cleaned (Cleaning batch number: APSLY50H-V1-CL-25001). Following step 4.3.4.1 of ECR (APSLY50HV1.ECR.E6), CIP cleaning was performed on the freeze dryer (FZ-V209). At 02/15/2025 01:48, an alarm (alarm code 30100, freeze dryer emergency stop triggered) appeared on the equipment HMI and the equipment stopped. The equipment could not be successfully reset, causing the CIP process to be unable to complete the automatic cleaning program. (Alarm message attached)
When the anomaly occurred, V1 line personnel also found that the stand-by filling machine (FL-V203), capping machine (CM-V205), and automatic feeding and discharging system (ALUS-V204) all triggered alarms simultaneously, and the control power failed and could not be restored. (Alarm message as attached)
Immediate Actions
Anomalous cleaning in APS (batch number: APSLY50H-V1-CL-25001) occurred on February 15, 2025 (Saturday). The anomaly was reported on February 17, 2025 (Monday), and the production supervisor assessed that it may affect the subsequent product FADP0121 EG1206A Drug Product 420mg for Infusion, 30mg/mL (batch number: FADP0121TR-V1-25002) process. Therefore, an immediate action (#55546) was initiated to evaluate the process impact and take corresponding measures.
The sterile water for injection product FADP0121 is used on the filling machine (FL-V203) and the capping machine (CM-V205). However, this anomaly caused the filling machine (FL-V203) and the capping machine (CM-V205) to trigger alarms simultaneously and failed to recover, resulting in the equipment being unable to operate normally and subsequent batch production.
Products FADP0121 (Lot: FADP0121TR-V1-25002) used in the DS have begun room temperature thawing in the thawing room (RM-V221) at 20:30 on 02/14/2025. It is expected to be completed by 20:30 on 02/18/2025, completing a 96-hour thawing process. Although it has not yet reached the end point of DS thawing, it may be affected by equipment abnormalities, so it cannot be put into production activities as scheduled.
It is understood that the customer discussed with the DS regarding the stability of the DS between 2 and 8 degrees Celsius. It was determined that the DS could be stored between 2 and 8 degrees Celsius for 6 months. As a result, it was decided that the 6 bags of DS would be thawed at room temperature for 96 hours (02/18/2025 20:30) and then moved to the 2-8 degrees Celsius refrigerator (CE-V264) located in the rewarming room (RM-V221). The batch record FADP0121TR-V1-25002 should be annotated with the abnormal code and the time of completion of the transfer of the 6 bags of DS to the 2-8 degrees Celsius refrigerator.
After thawing at room temperature in the DS room on 2025-02-18 20:30, all 6 DS bags were moved to the thawing room (RM-V221) and stored in the 2-8℃ refrigerator (CE-V264). The transfer was completed at 20:37 on 2025-02-18, and the batch record FADP0121TR-V1-25002 noted this exception number.
Edition：1
Investigation
The V1 production line at the APV was charged with batch APVLY5OH-V1-25001 on 02/15/2025 according to APSLY50HV1.ECR.E6 after cleaning (cleaning batch number: APSLY5OH-V1-CL-25001). When following ECR step 4.3.4 to perform CIP of the freeze dryer FZ-V209 at 02/15/2025 01:48, the HMI screen of the equipment popped up an abnormal alarm (alarm code 30100, freeze dryer emergency stop triggered). The alarm could not be restored, resulting in the CIP process being interrupted and the automatic cleaning process could not be completed.Freeze dryerCIPThe program has a total of5Cleaning procedures, which can be divided into2Times of washing into drainCleaning and3Times of circulation cleaning,(washing into drain)When the program is aborted, the freeze dryer has completed the second wash into drain.
At the time of the abnormality, the production line personnel also found that the filling machine FL‑V203, the capping machine CM‑V205, and the automatic feeding and discharging system ALUS‑V204 all triggered alarms at the same time and could not be restored (alarm history, please refer to Appendix 1). The alarm messages are summarized as follows:
|
|
|
|
|
|
|
|
At the evening of 2/15/2025, the safety controller automatically returned to normal operation.
The abnormal alarms on HMI can be eliminated and restored at this time.
However, at the evening of 2/15/2025, 2:15, four devices triggered alarms simultaneously (the alarms were the same as before) and could not be restored (alarm history please refer to Attachment 2).
It was not until around 10:00 on 2/17/2025 that the safety controller automatically recovered again, and the alarms on HMI can be eliminated.
During this period, the personnel did not take any action on the safety controller, indicating that the safety controller is in a very unstable state.
The alarm indicates two main categories: first, the safety controller malfunctioned/ did not start; second, the emergency stop was triggered. Since only the freeze dryer was performing a cleaning operation and the other three devices were in standby mode, there was no action on any of the four devices that could have triggered the emergency stop. Therefore, the reason for the safety controller malfunction/ not starting was further investigated.Alarm code 31039: Safety controller failure,
Alarm code 31032 indicates that the safety controller is not activated.
According to the alarm message on HMI, the maintenance personnel checked the safety controllers of the four devices on 02/15/2025. The status code displayed on the safety controller is ER18.
According to the manufacturer's operation manual, there are two ways to recover from the fault:
1) Self-recovery after 3 minutes.
2) Press the K point on the safety controller to recover.
However, after the maintenance personnel followed the instructions in the document, the recovery still failed. Therefore, it was reported to the production line manager and the equipment agent, hoping to seek help from the equipment manufacturer.
While waiting for the factory connection to be processed, the V1 production line personnel and engineers cross-checked the abnormal alarms and found that only the filling machine FL-V203 and the freeze dryer FZ-V209 triggered their own emergency stop alarms, causing the capping machine CM-V205 and the automatic feeding and discharging system ALUS-V204 to trigger their own emergency stop alarms due to the emergency stop of other equipment, which is not the emergency stop alarm of the equipment itself. Therefore, it is preliminarily ruled out that the safety controller of the capping machine CM-V205 or the automatic feeding and discharging system ALUS-V204 is abnormal.While the freeze dryer FZ-V209 did not issue an alarm for safety controller failure, the safety controllers of these four devices were originally designed as a series circuit by the manufacturer. Therefore, it is inferred that the safety controller of the filling machine may be abnormal, which caused the freeze dryer to be powered off and stopped urgently. After repeated tests by the subsequent production line personnel, it was found that as long as the safety controller of the filling machine resumes normal operation, the other 3 devices can be restored.
The production line personnel also tested various situations that might trigger the safety controller alarm, such as the intervention of the isolator on the safety light curtain and the opening and closing sensor of the isolator compartment door. The reason for this is to rule out the possibility that the alarm was caused by an abnormality in the safety device of the equipment itself. The production line supervisor also confirmed that there was no personnel error that triggered the emergency stop alarm, and if there was a human error that triggered the emergency stop device of one of the equipment, it would not cause the emergency stop alarm to jump out of the four equipment at the same time.
The original factory technician was connected remotely on 2/17/2025 to find the possible cause of the anomaly. At that time, the safety controller was functioning normally, and the original engineer did not find any abnormalities in the equipment, which means that when the safety controller is functioning normally, all four devices are functioning normally, and there is no equipment abnormality.
The original technician decoded the status code ER18 displayed on the safety controller as an error in analog input, indicating that the current value of the analog input is outside the allowable range or there is a problem with the signal processing. The engineering staff also conducted voltage measurements and signal continuity tests on the output and input signals of the safety controller (the test items are detailed in Attachment 3_Safety Controller Signal Measurement Confirmation Results). , To check if there is any communication anomaly. The test results show that the output and input signals are transmitted normally, and there is no communication anomaly or voltage anomaly. It is judged that it is not caused by communication anomaly, and the anomaly should be caused by the safety controller itself.
The next steps will involve further investigation into people, machines, materials, methods, environment, and measurement. The results of the investigation and the risk assessment for subsequent production activities are summarized below:
People
(Men)
The abnormality occurred during the CIP cleaning process of the freeze dryer (FZ-V209). The CIP cleaning process of the freeze dryer (FZ-V209) is an automatic process, and the V1 production line personnel have all completed the education and training according to the OPTIMAL freeze dryer standard operating procedures (Training Record #010325-009, 010225-010, 010225-008, 010325-010, 010325-011, 010225-011) and the APSLY50HV1.ECR.E6 education and training (Training Record #020825-009). The production operators followed the ECR to perform the cleaning work after the batch and operated the equipment according to the existing procedures. The abnormality was not caused by parameter setting errors or personnel not following the documents. It was also not caused by personnel manually operating the cleaning process abnormally. The abnormality alarm content displayed as a safety controller abnormality, but after confirmation by the production line supervisor, there was no abnormal operation by personnel at the time of the abnormality, triggering the safety alarm. Therefore, after evaluation, the risk of personnel factors causing this abnormality is low.
Machine (Machine)
Immersive Translate
The original technician performed a remote connection on 2025-02-17 to investigate the possible causes of the anomaly. The safety controller was in a normal recovery state at the time, and the original engineer did not find any abnormal equipment. This means that when the safety controller is operating normally, all four devices are operating normally, and there are no abnormal equipment problems.
The original technician interpreted the status code ER18 displayed on the safety controller as an analog input error, indicating that the current value of the analog input is outside the allowable range or there is a problem with the signal processing. However, after the engineer checked the safety controller output or input signal, there was no communication anomaly. Therefore, it is speculated that one of the safety controllers is abnormal, affecting the safety controllers of the four connected devices, causing the four devices to jump out of the same alarm at the same time, resulting in the equipment being unable to operate.
V1 The production line personnel and engineers conducted a comparison of abnormal alarms and found that only the filling machine FL-V203 and the freeze dryer FZ-V209 triggered emergency stop alarms themselves, while the capping machine CM-V205 and the automatic loading and unloading system ALUS-V204 were both triggered by other equipment emergency stop alarms, causing them to be linked. The initial cause is the safety controller of the filling machine FL-V203, the freeze dryer FZ-V209, the capping machine CM-V205 and the automatic loading and unloading system ALUS-V204.
Although the freeze dryer FZ-V209 has not issued an alarm that the safety controller has failed, the safety controllers of these four devices are originally designed by the manufacturer as a series circuit. Therefore, it is speculated that the filling machine's safety controller is abnormal, which caused the freeze dryer to power off and emergency stop. Subsequently, the production line personnel tested repeatedly and found that as long as the safety controller of the filling machine recovers to normal operation, the other 3 devices can be resumed.
The charging line control power is converted from DC power (Uninterruptible Power Supply) system, where the DC power system has a voltage stabilization design, so there will be no voltage drop problem. Therefore, the safety controller is excluded because the abnormal signal is caused by unstable voltage;
The engineer also confirmed that all signal lines are directly connected to the equipment from the control panel, and there is no transfer in the middle, and the lines use lock-tooth connectors, which are less likely to have bad contact than terminal connectors, so it is excluded that the abnormal signal is caused by bad contact of the lines.
In summary, the engineers believe that the filling machine's safety controller is malfunctioning. However, because the safety controllers for the filling machine FL-V203, the capping machine CM-V205, the automatic loading and unloading system ALUS-V204, and the freeze dryer FZ-V209 are designed by the original manufacturer to be connected in series, the four machines have triggered abnormal emergency stop alarms that cannot be reset.
Material
This abnormality occurred on the V1 production line of APSLY50H-V1-CL-25001 during the execution of the FZ-V209 freeze-drying machine CIP cleaning. The equipment HMI jumped out of the abnormal alarm and could not be restored. CIP is an automatic cleaning program. The abnormal alarm is displayed as a safety controller failure. Therefore, the influencing factors of raw materials are considered to be irrelevant to this abnormality.
Law (Method)
The anomaly occurred on the V1 production line according to APSLY50H-V1-CL-25001, when the FZ-V209 freeze dryer was performing CIP cleaning. The equipment HMI displayed an abnormal alarm and could not be restored. The CIP is an automatic cleaning procedure with established procedures and parameters. The production operators followed the existing procedures to operate the equipment, and it was not caused by parameter settings errors or personnel not following the documents. Therefore, after evaluation, the risk of this anomaly being caused by methodological factors is low.
Environment (Environment)
This abnormality occurred during the cleaning of the FZ-V209 Freeze Dryer CIP by the V1 production line according to APSLY50H-V1-CL-25001. The equipment HMI triggered an abnormal alarm and could not be restored. The abnormal alarm was displayed as a safety controller failure. After checking by engineering personnel, it was confirmed that all signals related to the isolator were 正常輸出輸入. This abnormality is not caused by abnormal conditions inside the isolator. Therefore, the risk of environmental factors causing this abnormality is low.
Measurement (測)
"The instruments listed in the Freeze Dryer (FZ-V209) remain within their calibration validity period. This alarm is caused by the safety controller failure, not the instrument abnormality. Therefore, after evaluation, the measurement factors are irrelevant to this anomaly."
Conclusion of Investigation
Cross-check of abnormality alarms of four devices revealed that the lid closing machine CM-V205 and the automatic feeding and discharging system ALUS-V204 both triggered emergency stop alarms due to the emergency stop of other devices. It was not an emergency stop alarm of the equipment itself. Therefore, the safety controller of the lid closing machine CM-V205 or the automatic feeding and discharging system ALUS-V204 is preliminarily ruled out as the cause of the abnormality. Although the freeze dryer FZ-V209 did not issue an alarm for safety controller failure, the safety controllers of these four devices are designed as a series circuit by the manufacturer. Therefore, it is inferred that the safety controller of the filling machine may be abnormal, causing the freeze dryer to lose power and stop urgently.
The original engineers remotely accessed the system on 02/17/2025 to troubleshoot potential abnormalities. The safety controller was operational at the time, and the engineers detected no equipment anomalies. This indicates that when the safety controller functioned normally, all four devices operated correctly with no abnormalities. They interpreted the status code **ER18** displayed on the safety controller as an error in analog input, signifying that the present analog input exceeded permissible bounds or exhibited a signal processing issue. However, following the engineers' inspection of safety controller output and input signals, no communication irregularities were discovered. Consequently, the engineers deduced that an anomaly in one safety controller may have impacted the linked controllers of the four devices, simultaneously triggering identical alerts on all devices, leading to equipment downtime and non-operational status.
The engineers believe that the abnormal situation is likely caused by a malfunctioning safety controller on the filling machine. However, due to the fact that the safety controllers of the filling machine (FL-V203), capping machine (CM-V205), automatic feeding system (ALUS-V204), and freeze dryer (FZ-V209) are designed by the manufacturer to be connected in series, the malfunction has triggered an emergency stop alarm on all four machines, which cannot be reset.
Root Cause analysis
The possible cause is an abnormality in the safety controller of the filling machine, but because the filling machine FL-V203, capping machine CM-V205, automatic material handling system ALUS-V204, and freeze dryer FZ-V209, the original design of the safety controller of the four devices is a series circuit, so the freeze dryer will jump out of the emergency stop trigger abnormal alarm, and cannot be restored, cannot complete the equipment automatic cleaning procedure.
Impact analysis
**Product quality impact**
Immersive Translate
50H Freeze-dried vial aseptic process simulation (filling batch: APSLY50H-V1-2 5001) as a routine annual revalidation activity. This anomaly occurred during the cleaning period after the production of the batch, not during the product production activity. Therefore, it has no direct impact on the quality of any product.
Batch filling APSLY50H-V1-25001 batch after cleaning ( cleaning batch number: APSLY50H-V1-CL-25001), according to ECR step 4.3.4 for FZ-V209 freeze dryer CIP cleaning, due to the equipment HMI abnormal alarm and cannot be restored, resulting in CIP program being aborted. At 02/17/2025 10:00, the safety controller resumed automatically, and the alarm on the HMI can be cleared. Then, the ECR procedure is executed again to perform the FZ-V209 freeze dryer CIP complete cleaning procedure. Although the first CIP cleaning of FZ-V209 was interrupted due to an abnormality, the freeze dryer is not directly contacted by the culture medium, and the APS process did not actually perform the freeze-drying process. The freeze dryer also started the CIP cleaning procedure within 24 hours after production, and after re-executing the FZ-V209 CIP procedure, according to ECR, TOC sampling and testing were also carried out, and it was confirmed that the specifications must be met. Therefore, after the FZ-V209 CIP cleaning was interrupted and cleaned again, it is evaluated that there is no impact on the equipment cleaning quality.
The impact on the device
After the original equipment manufacturer's technician remotely connected to the site, the production line and engineering personnel conducted cross-comparison and repeated testing, and it is concluded that the main reason is the abnormal filling machine safety controller. However, due to the original design of the filling machine FL-V203, capping machine CM-V205, automatic feeding and discharging system ALUS-V204, and freeze dryer FZ-V209, the safety controllers of the four devices are connected in series, which caused the four devices such as the freeze dryer to jump out of the emergency stop and trigger an abnormal alarm, and could not be restored. However, all four devices are functioning normally, and the abnormality is not caused by the devices themselves.
After the incident happened on 2/15/2025, the safety controller of the four devices resumed normal work at 6 pm on the same day. At this time, the abnormal alarm on HMI could be eliminated and restored. However, at 22:15 on the same day, four devices triggered the alarm again at the same time. The alarm is the same as before and it cannot be restored, until around 10:00 on 02/17/2025, the safety controllers automatically resume work again, and the alarms on the HMI can be eliminated. As observed so far, there is no such emergency stop trigger or safety controller failure alarm again. During this period, the staff did not take any action on the safety controller, indicating that the safety controller of FL-V203 is in a very unstable state. Therefore, a safety controller of the same brand and specification will be purchased for replacement. After replacement, it is necessary to confirm that all four devices are working properly before production activities can be started.
on the impact of the class zone
The possible root cause of this anomaly is not related to the level area, so it is estimated to have no impact.
(span id="0">Product batch impact/Scope(Date range)
This anomaly occurred during the batch post-cleaning of the APS requalification activity that is executed annually, and not during product production activities. Therefore, it does not directly impact any product batch.
Affect the batch of other products
The water needle filling line of the filling machine FL-V203 and the capping machine CM-V205 were shut down due to this abnormality. The control power supply failed, which in turn affected the start time of production of the next batch of products (FADP0121TR-V1-25002).
FADP0121TR-V1-25002 DShas beenarranged according to the schedule, andwasdeliveredfromWH on02/14/2025 20:30, and placed in thetempering room(RM-V221)for96hours of thawing and tempering, so the root cause of the anomalywas notfully identifiedon02/17/2025.With the customer's consent, based on the stability data of DS provided by the customer at 2-8oC for 6 months, the DS was thawed on 02/18/2025 20:30, after 96 hours of thawing, it was moved to the 2-8oC refrigerator CE-V264 for temporary storage on 02/18/2025 20:37. Production activities will resume after confirmation that the equipment operation has resumed.
Process Validation/Registration Document Impact
This exception occurred during the annual routine APS revalidation activity, and the equipment was cleaned in batches thereafter. This exception will not affect or cause changes to critical process parameters, so it is assessed that it will not affect process validation.
This anomaly occurred during the post-batch equipment cleaning process of the annual routine re-validation activity, which is not related to regulatory registration. Therefore, it is assessed that it will not affect the registration documents.
Stability/Re-Inspection Evaluation
This anomaly occurred during a batch equipment cleaning operation following the annual routine APS re-verification activity. This incident did not involve product production activities, and there is no impact on product quality. It will not affect the stability/re-verification period and is determined to have no impact.
Customer Notice Evaluation
This is a batch after-verification cleaning of equipment performed during the routine APS in the factory, but due to an anomaly, the next batch of water probes (FADP0121TR-V1-25002) DS has already begun the defrosting process, so we need to notify customers.After evaluation, the root cause of this anomaly is not fully confirmed for the batch start date of F ADP0121 (February 17, 2025). With the customer's consent, based on the condition that DS provides 6 months of stability data at 2-8°C, DS thawed at 20:30 on February 18, 2025, and was moved to a 2-8°C refrigerator CE-V264 at 20:37 on February 18, 2025, 96 hours later. After confirmation of equipment operation recovery, production activities will be restarted.
Risk Analysis
Based on the abnormal investigation procedures in SOP003.E19, use the risk index (Risk Prioritization Number) for risk assessment:
Severity(S): 3
The abnormal event is caused by the abnormal safety controller of the filling machine, but the filling machine FL-V203, the capping machine CM-V205, the automatic feeding and discharging system ALUS-V204 and the freeze dryer FZ-V209 have four safety controllers. The original design of the equipment is a series circuit, so the freeze dryer jumps out of the emergency stop trigger alarm and cannot be restored. The reason leads to APSLY50H-V1-CL-25001 batch step 4.3.4, which was performed after cleaning FZ-V209 freeze dryer CIP cleaning, so that the CIP program could not complete the automatic cleaning procedure. The abnormal situation occurred during the factory routine APS reconfirmation activity. During the post-production batch cleaning period, it is not a product production activity. Therefore, it does not directly affect any product quality. After the equipment resumes normal operation, the freeze dryer FZ-V209 re-executed the CIP program, and the sample was taken according to the ECR execution TOC, and it was confirmed that the specification must be met. Therefore, FZ-V209 CIP was cleaned again after the cleaning was interrupted, and it was evaluated that it had no impact on the equipment cleaning quality.
This anomaly occurred during the routine APS revalidation activity within the plant. However, since the next water needle batch (FADP0121TR-V1-25002) had already started the thawing process when the anomaly occurred, the root cause of the anomaly in the FADP0121 batch could not be fully confirmed on the start date of production (02/17/2025). With the consent of the customer, and based on the condition that the DS provided by the customer has 6 months of stability data at 2-8°C, the DS was thawed at 20:30 on 02/18/2025 and moved to a 2-8°C refrigerator (CE-V264) at 20:37 on 02/18/2025 after 96 hours. After confirming that the equipment operation has resumed, production activities will be restarted. As the production batch has not yet entered the production stage, the risk of affecting the quality of the batch is assessed as low.
Based on the above, the severity score is 3.
Occurrences (O): 1
The translated paragraph in English is:
View all production batches that have been executed on the V1 production line (approximately 27 batches have been produced since statistics from 2022), the same abnormal phenomenon has occurred only once, so the occurrence rate score is 1.
Detection (D): 4
Okay, here's
V1 production line personnel will conduct equipment condition inspections before production, and there will also be CIP/SIP or VHP operations before official filling, such as:
If the safety controller is abnormal, there is a chance to detect it early and reduce the sudden situation in the production process, thus reducing the direct impact on the product.
Therefore, the detection degree score is 4.
Risk Index (RPN): S*O*D = 3*1*4=12, which is considered Minor according to the evaluation criteria
Corrective action
Immediately replace the filling machine FL-V203 safety controller with the new one. After replacement, check whether the four devices are powered normally and conduct safety alarm trigger tests on the four devices. After confirming that the safety mechanisms of each device are working properly, the equipment can be used for GMP production activities.
Observe if the equipment still experiences the same abnormalities within 90 days after the completion of the CA
Prevention measures
To purchase spare parts for the safety controller, in order to shorten the time of production process delay when the same anomaly occurs in the future.
NO CAP
A child justification
Safety controllers are long lead time components, and their internal electronic failures are unpredictable. If a failure occurs, it may cause prolonged downtime, affecting production capacity and operational efficiency, thus increasing production risks and costs.
To ensure the stable operation of the equipment and shorten the repair time, the engineering department plans to establish a spare parts inventory and allocate a project budget for three consecutive years to ensure the uninterrupted supply of safety controllers for V2, V3, and D Line equipment, maintain smooth production, and achieve the best balance between equipment management and financial operations.
