Quality Assurance Criteria Review and Approach Documents Suggested Lines of Inquiry

• Has the contractor assigned a senior management position for developing, implementing, assessing, maintaining and improving a Quality Assurance Program (QAP) that meets general requirements of Paragraph 2 delineated in DOE O 414.1C, Quality Assurance, 6-17-05, Attachment 2 (Contractor Requirements Document (CRD))? [The point is, does the Quality Assurance Manager report at a level of management that enables positive management action to be applied on issues related to quality? If not, look for examples that may be used to support an argument that inadequacies of management attention to quality is resulting in not achieving the desired level of quality performance.]
• Does the contractor's QAP describe how the "graded approach" concept is to be applied to work performed? [The question here is if guidance has been provided in the contractor's QAP for applying quality principles to work control applications for the various severity levels of importance.]
• Does the contractor's QAP include requirements for preventing suspect/counterfeit items (S/CI) from infiltrating into items accepted for use? [The contractor's plan for the control of S/CI items should be researched beginning with direction provided by senior management and followed through the actual application of the directives in facility work controls (procedures) that workers follow in performing their work. Worker performance is an excellent process for performing an observation of how work is done. In the course of observing work being performed, it is appropriate to question workers on their understanding of items that may have been identified as S/CI materials.]
• Does the contractor's QAP address quality requirements to be applied for safety software? How are quality requirements for safety software communicated to workers (orally, procedurally, training, or other means)?
• Has the contractor authorized the use of national or international consensus standards? If consensus standards are utilized, is there justification provided regarding their applicability and how they are to be or will be applied? [Standards may be adopted through regulation, code, contract, QAP, or procedure. Once a standard is adopted, compliance is required and is not voluntary.]
• Has the contractor integrated other accepted (per Order 414.1C) quality management system requirements defined in DOE directives and external requirements? Examples of documents that provide acceptable quality requirements include:
  
  
  • DOE P 450.4, Safety Management System Policy, dated 10-15-96;
  • DOE P 450.5, Line Environment, Safety and Health Oversight, dated 06-26-97;
  • NNSA Quality Management Policy, QC-1 (quality management system for the nuclear weapons complex and weapons-related activities);
  • DOE/RW-0333P, DOE Office of Civilian Radioactive Waste Management, Quality Assurance Requirements and Description; and
  • DOE/CBFO-94-1012, DOE Carlsbad Field Office, Quality Assurance Program Description, (for the Waste Isolation Pilot Plant and related activities).


• Are changes to the contractor's QAP processed as follows:
  
  
  • A copy is submitted to DOE for approval before beginning work under a DOE contract?
  • The QAP is being implemented as approved and modified by DOE?
  • Third party certification, if any, affecting the QAP is indicated in the submittal to DOE?
  • Enhancements required by the current CRD are addressed through revisions to the existing QAP that was developed based on previous versions of DOE O 414.1(A and B)?
  • Has the contractor worked to QAP revisions that were not responded to by DOE in the 90 calendar day requirement? Are there any potential issues that DOE negligence in responding may not have identified?


• Are QAP changes that were made in the previous year submitted per the schedule time for DOE review and approval? Does the documentation provided by the contractor delineate the changes, reasons for the changes, and a basis for concluding the revised QAP continues to comply with requirements of the existing CRD?
• Does the contractor include a consideration of the QA guidance provided via DOE guidance documents in developing and implementing the QAP? Examples of DOE guidance documents include:
  
  
  • DOE G 414.1-1, Management and Independent Assessments Guide for Use with 10 CFR, Part 830, Subpart A, and DOE O 414.1C, Quality Assurance; DOE M 450.4-1, Integrated Safety Management System Manual; and DOE O 226.1A, Implementation of Department of Energy Oversight Policy, dated 9-27-07.
  • DOE G 414.1-2A, Quality Assurance Management System Guide for Use with 10 CFR 830 Subpart A Quality Assurance Requirements and DOE O 414.1C, Quality Assurance, dated 6-17-05.
  • DOE G 414.1-3, Suspect/Counterfeit Items Guide for Use with 10 CFR 830 Subpart A Quality Assurance Requirements and DOE O 414.1B, Quality Assurance, dated 11-03-04.
  • DOE G 414.1-4, Safety Software Guide for use with 10 CFR 830 Subpart A, Quality Assurance Requirements, and DOE O 414.1C, Quality Assurance, dated 6-17-05.

Performance Objective 2: Contractor Quality Assurance Program Implementation

Lines of Inquiry

• Has a Quality Assurance Program (QAP) description document been developed? If not, why? How is quality being assured without a document that describes the contractors program?
• Does the QAP describe the management processes that the contractor applies in the daily management of work? If not covered in the QAP, how does the contractor communicate management processes that ensure workers know what is expected from their performance of work?
• Does the QAP include a description of the organization structure, functional responsibilities, levels of authority, and identify interfaces for those managing, performing and assessing work? If not covered in a QAP document, how is the information communicated to workers and do workers have a clear understanding of their roles and responsibilities?
• Does the QAP clearly identify a senior level management commitment to quality and safety and that every component and employee of the organization is included in the scope of the quality management system? If not covered in the QAP, where and how does management communicate this commitment?
• Did senior management (defined as the top manager of the program or project) approve the QAP? If not approved by senior management, does the approval authority have sufficient support from senior management to ensure quality and safety issues are not compromised should conflict develop between competing interests that could result in compromising quality/safety? A review of periodic performance reports that address a lack of production progress or work completion delays may serve as an indication of quality weaknesses such as the amount of rework that had to be performed, or inadequate direction given to workers, or inadequate training of workers. The reviewer should strive to identify examples and sources of weaknesses.
• Has the contractor established a process that ensures persons responsible for managing and performing activities possess the appropriate level of experience, knowledge, skills and abilities that are commensurate with their responsibilities? A review of worker skill requirements against the individual qualifications of assigned workers should be made to assess the adequacy of the contractors program.
• Does the QAP include a description of management processes to be applied for planning, scheduling and selection of resources to be allocated for work? If not described in the QAP, is this covered in other documents? Identify and review documents that communicate requirements for planning, scheduling and selection of resources for performing work.
• Does the contractor's quality management program include a need for conducting self-assessments or management assessments, and independent assessments as defined in the DOE Order 414.1? If not performed, what is management's position for not having performed assessments? Do managers and workers clearly understand what is expected? If so, how is this being performed?
• Does contractor senior management monitor performance trends for work that would indicate the effectiveness of quality management controls that have been implemented? Ask a senior manager to describe how he/she monitors performance and have him/her show you evidence of feedback that supports conclusions that are being drawn.
• Does the management process encourage and include worker participation in identifying opportunities for quality/safety improvements? Ask workers if their expertise is sought on work processes to perform applicable work, and if not, what is their perception of the adequacy of the direction they are given?

2.2 Personnel Training and Qualification: Contractor worker qualification and training processes are defined and implemented to ensure personnel achieve and maintain the required capabilities to perform assigned work.

Lines of Inquiry:

• Has the contractor established qualification requirements for workers? Documents that establish worker qualification requirements should be reviewed to establish the adequacy of the contractor's program. Documents include contracts, task description documents, training requirement documents and other means that the contractor has implemented and uses. The review should include how requirements are passed down to subcontractors and for persons who serve as Subject Matter Experts (SME).
• Has the contractor made provisions and committed resources that facilitate the training needs and qualification of personnel? Are responsibilities and authority for persons who provide training clearly defined and understood and are they held accountable for carrying out their responsibilities? Does the contractor use indicators of training performance to assess improvements needed in training? Are training programs systematically evaluated and revised to maintain and improve required skills and knowledge?
• Does management assure that persons selected for positions through transfer or hire have the necessary qualifications that are appropriate for performing assigned work? Select samples from the contractor's human resource files and track the contractor's activities that were followed in getting them prepared for their work assignments.
• Are job/worker skills proficiency requirements defined and worker training needs developed based on defined requirements? Review documents that perform this function to establish if they are consistent with the need to ensure worker skill proficiency is properly established and workers possess the level of proficiency desired by management for performing work.
• Does management solicit input from workers regarding the adequacy of training that is being provided? Is feedback from job performance used to help evaluate and refine training programs? Interviews of managers and workers should result in a consistent and compatible understanding of what is desired and whether results are adequately achieved.
• Do training modules include consideration of safety, emergency plans, security and operations information that is necessary for personnel to adequately prepare for and perform assigned duties? Review contractor problem reports to determine if there are weaknesses in the contractor's program for preparing workers to perform assigned duties?
• Does training modules include a message about the organization's mission, vision, goals and the importance of all workers and their contribution to the overall achievement? Is this communicated to all levels of workers within the contractor's organization? If not, are there problems that could be cited as potential weaknesses in this aspect of the contractor's program.
• Does management assure that training plan content is current and reflects the latest information regarding the site, facility, and organization procedures and controls; technical and professional references; and past organization/industry experience and applicable lessons-learned? If not, what is the contractor's management position for not ensuring training plan content is current?
• Have training record needs been identified and are they maintained current? A review of a sample of training records should be performed to determine if they are up-to-date and that current training needs are being implemented in a pro-active manner.

Lines of Inquiry:

• Does the contractor have in place a means for identification of quality problems? It is suggested the contractor's QAP be reviewed to see what has been defined. The next approach is to ask various managers to describe the contractor's program for quality improvement. Finally, interviews with workers should be performed to establish their level of understanding of the contractor's quality improvement program. Observations of work situations may enable identifying manager/worker commitment to quality improvement processes and opportunities.
• Are contractor personnel at all levels committed to quality performance? Identify how commitment to quality performance is communicated and applied through reviews of work control procedures, and training module information. Interviews with workers enable gaining worker perception and understanding of commitments to quality performance.
• Are quality audits and surveillances scheduled based on the importance of the activity, past performance, and suspected weak areas? Are they timed to identify problems early in a process? Interviews with the QA Manager and supervisors are sources of how audits and surveillances are scheduled and conducted. A representative sample of audits and surveillance reports should be reviewed and assessed regarding their adequacy for identifying precursors to problems.
• Are the results of quality and surveillance audits documented and evaluated to allow early detection and correction of performance problems? Are quality problems being identified; evaluated and their causes and significance determined; and are management disposition actions effectively applied?
• Does the contractor emphasize at all worker levels the desired approach for improving quality is the "prevention" of quality problems?
• When improvements are determined necessary, does management balance quality/safety requirements with mission priorities? If not, what seems to be the predominate tendency regarding management decisions?
• Does contractor management encourage employee awareness and participation in planning, developing, exploring and implementing new ideas for improving quality in products, processes, and services?
• Has the contractor established a program to analyze quality related information from various internal and external sources to identify improvement opportunities in the quality management system, processes, items, products, or services? Examples of this should be reviewed to determine the adequacy of quality improvement initiatives.
• Has the contractor implemented a tracking system or program that enables effective follow-up on the adequacy of actions to improve quality? The reviewer should identify examples of where this is applied and assess the adequacy of implementation.
• Has the contractor established a program that ensures items, services, and processes that do not meet established requirements are identified, isolated and controlled to prevent their being applied during work processes? This should be explored with both management and workers to ensure a consistent understanding exist as to what is expected.
• Does the contractor's corrective action program include the identification of "root-causes" to quality and safety problems to prevent recurrence? If not, ask the contractor's QA manager for an explanation of what is being done to ensure causes to problems are established and changes to management controls are effective in preventing their recurrence.
• Has the contractor implemented a "Lessons-Learned" program that utilizes feedback from within and other related organizations to assist efforts in minimizing or preventing quality problems? If not, ask the contractor's QA manager for an explanation of what is being done to ensure causes to problems are established and changes to management are effective in preventing their recurrence.
• Does the contractor track through resolution quality problems identified as Type "A" Accident events, Price-Anderson enforcement events, and Inspector General identified issues? If not, ask the contractor's QA manager for an explanation of what is being done to ensure causes to problems are established and changes to management are effective in preventing their recurrence.

Documents and Records Lines of Inquiry:

• Has the contractor fully implemented a comprehensive document and records control management system? Is there a document or set of documents applicable to the control of documents and records and are workers consistent in their following the guidance provided?
• Are documents that control work processes, specify requirements, or establish designs prepared, reviewed, approved, issued, and revised in a controlled manner? Are these documents readily available to facility personnel having responsibilities for maintenance and operation of facility safety related and non-safety related systems? It is suggested that the reviewer assess the various groups (design, maintenance, and operations organizations) to see if there are procedures that establish a set of controls for development of documents and records and the relationship and interface is clearly defined with the group that performs a document control/records management service.
• Does the document control management system ensure that safety and quality management requirements are appropriately documented and communicated by ensuring workers have the proper documents for performing work? Does the document control management system support maintenance of facility configuration control through a process of controlled document reviews for proposed changes to the facility configuration? [The point here is that the document control function should reflect the applicable version of documents that workers should be using to perform their assigned tasks.]
• Are facility documents maintained current to reflect the actual facility configuration and current design requirements? Maintained current implies that design requirements for facilities are correctly reflected in documents, and that facilities reflect design requirements as specified in approved design documents.
• Have records requirements been defined and are persons who are responsible for implementing the requirements been trained on the requirements? How are records defined and who is responsible for identifying documents that become records? Is this process clearly defined and have persons performing documents/records management functions been fully trained on their functions?
• Do records adequately support technical, regulatory and enforcement decisions and provide evidence that work was correctly performed? The reviewer should select a sample of documents and follow the established management process to see that records do accomplish this function.
• Are there adequate means and controls for the development and preservation of records for all record forms being maintained (e.g., electronic, written, printed, microfilm, photographs, optical disks, etc.)? The reviewer should select examples of the various record types and trace the established process to examine if the system is working.
• Are there schedules for records retention and disposition and are they consistent with requirements of DOE O 200.1, Information Management Program, dated 9-3-96?
• Is the hardware and software used to create and store records being maintained to enable records to be readily retrievable? The reviewer should select examples and test the process for retrieving records.

Lines of Inquiry

• Are contractor work control procedures and standards that are used by workers consistent with accepted technical standards, administrative controls, and hazard controls that have been adopted to meet regulatory requirements? The reviewer should select and review a representative sample of procedures and standards used by workers to establish that the contractor's management processes do ensure regulatory requirements are being complied with.
• Has the contractor implemented a management process that results in line management and workers cooperatively reviewing work processes that can be improved? This may include feedback from prior work as well as supervisor-worker critiques that follow the implementation of completed work. The reviewer should establish if there are activities scheduled during the facility/site review and seek to observe the actual conduct of a supervisor-worker review session that addresses improvements in their processes. Further, the reviewer should ask for reports that document the results of such actions. The reviewer may wish to prepare a report that documents the results of the observation which could be informative and helpful in identifying both positive and negative aspects of the process.
• Does contractor management solicit and encourage input from workers regarding work processes? Worker attitude and moral can be an indicator of possible deficiencies in the management/worker relationship and its ability to generate a "team approach" to the performance of quality work. The reviewer should look at existing work control procedures being applied and solicit comments from workers regarding the clarity of what is expected as documented in the procedure, problems perceived by the workers, and their input as to their role, if any, in the actual planning for performing the work.
• Does contractor management communicate an expectation of worker accountability for quality and safety in performance of work? Do workers understand management's expectations regarding their performance? Whatever their response, the reviewer should establish the basis for worker responses, either positive or negative, to questions asked by the reviewer?
• Does contractor management ensure the following information is communicated to workers prior to the beginning of work:
  
  
  • Customer and data requirements for the work and final product?
  • Hazards associated with the work to be performed?
  • Safety, administrative, technical, environmental, and quality controls to be applied during the work?
  • Technical standards to be applied?
  • Acceptance criteria applicable to the work?
  • Procedures to be applied for verification of completed work?


• If elements of the above are missing, determine the worker's perception of what is expected and the basis for the perception?
• Are work instructions or means used to define the work processes documented and controlled? This concern focus' on the adequacy of the contractor's document control and work control practices for assuring workers have the correct documents to perform work.
• Does the contractor follow a "graded approach" for the development of work instruction detail that is commensurate with the complexity and the importance of the work? This inquiry is getting at the contractor's methods for prioritizing work from a perspective of quality/safety, and the depth of detail regarding the controls to be applied.
• Do workers indicate their complete understanding of controls that are applied to the work and are there provisions for them to document their understanding? The intent here is to ensure that management communicates the level of importance expected for worker adherence to controls imposed on the worker, and that the worker clearly understands managements concern regarding their attention to the detail necessary to perform the work. This can be explored through reviews of documents provided by management and interviews of workers to assess the adequacy of their understanding of what is required. Confirmation of worker understanding should be weighed against management's expectations that are placed on workers. Interviews of both management and workers may be necessary to arrive at a conclusion of the adequacy of this process.

Lines of Inquiry:

• Has the contractor implemented controls that prevent workers from using incorrect or defective items? If not specifically defined, the reviewer should determine the contractor's process for ensuring workers do not access or use incorrect or defective items.
• Does the contractor have a program for identifying and controlling suspect/counterfeit items?
• Does the contractor have a program for controlling and maintaining items?
• Are items and materials maintained and stored to prevent their damage, loss or deterioration?
• Does the contractor's program for the control of items include supplier/manufacturer controls to ensure that suppliers use quality materials for the manufacture of items?
• Does the contractor's materials control program ensure that equipment/tools used for process monitoring or data collection are calibrated, maintained, and properly secured?

The reviewer may find it advantageous to perform a thorough review of the contractor's management processes for controlling materials. This should start with the origination of the basis for the need, assessment of the controls applied for preparation and approval of specifications, assessment of processes required to be followed for the selection of suppliers/vendors, inspection controls placed on the manufacturer, receipt inspection controls that are applied, storage controls required for items when received at the site, maintenance controls such as the environment required when in storage, and the handling processes to ensure no damage during installation.

Design - Lines of Inquiry

• Has the contractor implemented a management process that enables the control of design inputs and outputs, design verification, design changes and configuration control? The contractor's design engineering management process controls should be reviewed to determine that design procedures are in place and are rigidly followed that control each design function.
• Does the design process ensure that sound engineering/scientific principles and applicable standards are utilized to design items and processes? The contractor's management system defines requirements for persons authorized to perform design engineering functions and the persons performing design functions fully meet the contractor's qualification requirements.
• Does design bases documents include all applicable requirements that are needed to develop detail design documents and for making changes to designs? This requirement is focused on design inputs. The contractor's design documentation should clearly establish what the requirements for each design are.
• Are design interfaces, technical and administrative, clearly defined and controlled?
• Are persons who are selected to perform design verification technically qualified to perform the work and are they independent of the design work they are assigned to verify?
• Does the contractor verify design products before they are approved and issued for implementation?
• Does the contractor use a "graded approach" in selecting the applicable design control requirements to be applied in developing designs?
• Does the contractor's "graded approach," include a consideration of safety functions such as the safety class and its significance to safety for structures, systems, and components?
• Does design output documentation provide acceptance, inspection, testing, and maintenance criteria for ensuring continuing reliability and performance of functional capability for items?
• Does the contractor's design organization have a process for verification of the adequacy of software that is used for supporting the development of designs?
• Do design records include documentation such as design inputs, calculations and analyses, engineering reports, design outputs, design changes, design verification documentation, and other supporting documentation that provides evidence that the design process was performed correctly?
• Are contractual requirements and customer expectations reflected in design input documentation?
• Do design output documents support other management processes such as dose and risk assessments, procurement, manufacturing, assembly, construction, testing, operation, inspection, maintenance, and decommissioning?
• Are design output documents maintained current such that they reflect the actual installed status or "as built" configuration for structures, systems, and components?
• Does the contractor's design verification program include a requirement for design reviews, alternative analyses, qualification testing, and peer reviews?
• If the contractor allows interim (or partial) design verification to occur, is there a set of management controls that supports work such as procurement, manufacture, construction, or testing?
• Does the contractor require design verification to be completed before reliance is placed on structures, systems, and components to perform their function?
• Does the contractor's design process include a verification process for design changes that result from temporary modifications, or when installed designs do not reflect the design output documents for the item and is dispositioned "use-as-is?" If not, how are discrepancies in items dispositioned when accepted items differ from requirements provided in design output documents?
• Has the contractor implemented special requirements for computer software that is used to originate or analyze design solutions that serve to mitigate potential accidents?

Lines of Inquiry

• Is the contractor's procurement management process commensurate with the importance for the end use of the purchased item or service? For example, are the procurement processes for safety-related structures, systems, and components graded based on their importance to safety and the protection of workers, public and environment?
• Are procurement processes planned and controlled to ensure that procured items and services meet established requirements and perform as specified; that prospective suppliers or evaluated and selected on the basis of established criteria; and only approved suppliers are utilized that have established a track record of continuous providing acceptable items or services?
• Does the procurement process ensure that suspect/counterfeit items are precluded from receipt? If not, does the contractor have a program for ensuring that items fully meet acceptance requirements at receipt?
• Does the contractor have a process in place for identifying suspect/counterfeit items, and does the process ensure total exclusion of such items from accepted items?
• Has the contractor's management controls for procurement been verified that they fully comply with DOE procurement and subcontracts through applicable DOE Orders, the Department of Energy Acquisition Regulation (DEAR) in 48 CFR subchapters A through H, and Federal Acquisition Regulation (FAR) in 48 CFR 970 et.seq?
• Does the existing procurement management system adequately meet end-user requirements and reflect requirements of the QA Rule and Order?
• Does the procurement process demonstrate how the contractor's responsibilities are satisfied for the criteria of 10 CFR 830.122 for nuclear facilities when subcontractors and suppliers perform work?
• Is potential or prospective supplier capabilities reviewed and established early in the design and procurement phases?
• Does the supplier qualification process include a review of the supplier's history for providing similar or identical services; an evaluation of certifications or registrations awarded by nationally accredited third parties; and an evaluation of qualitative and quantitative performance information provided by the supplier?
• Does the contractor follow a process for the pre-qualification of prospective suppliers of structures, systems, and components? Does the contractor's requirement for prospective suppliers include having a quality assurance program and that it is verified to conform to contract requirements. Further, are prospective suppliers required to demonstrate their capability to meet performance and schedule requirements?
• Does the contractor's review of suppliers include periodic follow-on reviews to verify and monitor that capabilities are continuing to be maintained and applied?
• Do contractor's procurement documents specify critical or important acceptance parameters and "hold-points" for item inspection? Do they include verification that the supplier has provided specified documentation and that items were not damaged during shipment? Do they require verification or testing of items prior to or following shipment?
• Does the contractor's acceptance program include provisions for supplier-generated documents be accepted through the procurement system and controlled by the end-user organization? Documents may include certificates of conformance, drawings, analyses, test reports, maintenance data, non-conformance documentation, approved changes, waivers, and accepted deviations.
• Does the contractor's procurement program include the purchase of commercial-grade items that are intended for use in nuclear safety or high-risk activities? If so, was the procurement in accordance with documented processes that utilized recognized consensus standards?
• Does the contractor engage in multi-site procurement practices? If so, have the sites agreed upon a set of requirements and responsibilities that govern quality requirements for procurements?

Lines of Inquiry

• Has the contractor established acceptance and performance criteria for inspecting and testing specified items, services, and processes?
• Does the contractor follow a rigid program for calibrating and maintaining equipment used for inspections and tests?
• Has the contractor specified technical qualification requirements for personnel selected to perform inspections and tests and are they empowered to access appropriate information and facilities to verify item acceptance?
• Are inspection and test personnel independent of activities being inspected and tested and do they have the freedom to report results of the inspection/tests?
• Are inspection/test results evaluated and verified by qualified personnel of the design organization to document that design requirements are satisfied?
• Does planning for inspection/tests include identification of characteristics to be examined; qualification requirements for persons performing the examinations; descriptions of examination methods to be applied, including equipment and calibration requirements; acceptance and rejection criteria; environmental conditions for the examination; shelf-life and maintenance requirements, if any; required safety measures to be applied; and mandatory hold points, when applicable?
• Does the contractor's inspection/tests acceptance program provide for identification and storage requirements for accepted items that ensures their correct application for the intended use?
• In the event of changed item acceptance requirement parameters, are accepted items reviewed by persons knowledgeable of design requirements to determine if re-inspection and testing of item(s) is (are) necessary?
• Is the contractor's Measuring and Test Equipment (M&TE) calibrated, maintained, and controlled using a documented process? Is the M&TE checked before each use to ensure it is of the proper type, range, accuracy, and precision, and is uniquely identified and traceable to its calibration data? Do contractor personnel work to procedures that establish requirements for testing, retesting, adjusting, and recalibrating M&TE? Are M&TE calibrated to traceable standards such as the National Institute of Standards and Technology or other nationally recognized standards when appropriate? Is M&TE use traceable to the accepted item?

Lines of Inquiry

• Has contractor management implemented a management assessment program?
• Are management assessments performed to determine the effectiveness of management processes versus expectations for performance? Can managers identify examples of assessments and improvements that resulted?
• Do contractor senior managers emphasize the importance of self-assessments and ensure that subordinate managers plan, schedule, conduct, evaluate and document results of self-assessments?
• Do managers view their participation in the assessment process as being essential to ensure their gaining a first-hand understanding of how well their management systems are functioning?
• Are managers aware of any problem(s) that may be hindering their organization from achieving their objectives; do they aggressively pursue corrective measures; and are personnel trained on changes that result from corrective actions?

Lines of Inquiry

• Has contractor senior management implemented a program of independent assessments for its organization's programs, projects, suppliers and subcontractors?
• Do managers plan, schedule and conduct independent assessments to measure item and service quality and the adequacy of work performance to promote improvement?
• Does the management structure ensure there is sufficient authority and freedom from line management to enable independent teams to effectively perform assessments?
• Are persons who conduct independent assessments technically qualified and knowledgeable in the areas they are assigned to assess and have had no previous responsibility for the work being assessed?
• Do independent assessments include an evaluation of the following:
  
  
  • Work performance and process effectiveness;
  • Quality of work products;
  • Compliance to established management system requirements;
  • Identification of abnormal performance and potential problems;
  • Opportunities for improvements; and
  • Documenting and reporting of assessment results?


• Are results of independent assessments analyzed to determine if problems are global to establish the appropriate remedial management action to be initiated for correcting problems?
• Are findings from assessments tracked to determine progress and effectiveness for correcting identified problems? Are results of the tracking activity communicated to senior management?
• Does the contractor use a "Lessons Learned" reporting mechanism to enable other management groups to benefit from identified issues?

Performance Objective 3: DOE Line Management Oversight

DOE line management has established and implemented effective oversight processes that enable evaluating the adequacy and effectiveness of contractor quality assurance management systems and DOE oversight processes.

Lines of Inquiry

• Has the DOE Office developed and effectively implemented quality assuring management programs and trained employees on the application of the programs? Do employees reflect a thorough understanding of what is expected of them; do they understand their role and responsibilities for quality and safety?
• Do DOE line managers communicate an attitude of responsibility and accountability for quality of work performed in their assigned areas? Do they ensure their subordinates are informed and trained on what is expected regarding performance of work? Does line management perceive any problems they are experiencing in obtaining a quality performance of staff?
• Does the applicable DOE QAP allow for the use of national or international consensus standards where practicable that is consistent with regulatory requirements? Are standards that have been authorized for use identified? [Examples of appropriate standards include ASME NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications (for nuclear related activities); ANSI/ISO/ASQ Q 9001-2000, Quality Management System; Requirements (for nonnuclear activities); and ANSI/ASQ Z 1.13, Quality Guidelines for Research, 1999 (for nonnuclear research activities).]
• Are standards other than those listed above being applied that were determined necessary to address unique/specific work activities? Are these standards consistent with contractual or regulatory requirements? [Note: These standards may be referred to "voluntary standards." Once a voluntary standard is adopted through regulation, code, contract, QAP or procedure, compliance with the standard is required and is not voluntary.]
• Has the DOE Office integrated, where applicable, contract or regulatory requirements, quality management system requirements as defined in DOE O 414.1C, Quality Assurance, the S/CI prevention process, the Corrective Action Management Program (CAMP), and Safety Software Quality Requirements with other quality or management system requirements in DOE directives and external requirements, including as applicable the following:
  
  
  • DOE P 450.4, Safety Management System Policy, dated 10-15-96;DOE P 450.5, Line Environment, Safety and Health Oversight, dated 06-26-97;
  • NNSA Quality Management Policy, QC-1, (quality management system for the nuclear weapons complex and weapons-related activities);
  • DOE/RW-0333P DOE Office of Civilian Radioactive Waste Management, Quality Assurance Requirements and Description; and
  • DOE/CBFO-94-1012, DOE Carlsbad Field Office, Quality Assurance Program Description, (for the Waste Isolation Pilot Plant and related activities)?


• When talking with DOE employees regarding quality issues, do they understand what is expected? Do employees have issues that need management attention? Do employees perceive that their experience and "know how" is encouraged for the benefit of quality? Do employees express a strong commitment to the principles of quality?
• Does Line Management communicate quality requirements effectively that ensures all working groups properly and fully understand their roles and responsibilities?
• Does line management utilize a "graded approach" in the application of quality measures to non-safety related activities? Is line management consistent in the implementation of the graded approach process? Do employees have concerns regarding the application of quality requirements and how their concerns flowed up to managers?
• Has senior management implemented an on-going process of audits and surveillances and is the process effective in the identification of quality issues? Are the persons who conduct audits and surveillance activities qualified and capable of identification of performance issues?
• Is the management tracking system for corrective actions effective in communicating progress in fixing quality problems? Do senior managers communicate interest in problems that have been identified and do they follow progress towards closure of issues?
• Is the process for analyzing problems effective in establishing the full extent of quality issues? Does line management periodically review past issues to determine if remedial action is continuing to correct management weaknesses?