EPSD uses artificial intelligence tools such as Claude and Claude Code. When determining the suitability of using an AI tool, we look at factors that help us understand the business process we want to improve or the problem to be solved and the applicability of AI tools to solve it.
For example:
- Is this a process we’ve previously tried to fix? What happened? Why did the fix not hold?
- Is a non-deterministic result acceptable?
- How bad is the damage (to us, to our customers, and to our stakeholders) if the AI tool produces the wrong, or an unexpected or invalid, response?
- Is it a process that lends itself well to AI tools, like repetitive, simple choices? Or does the problem require more nuanced analysis before making a decision?
- What access is required by the AI tool to our internal data and systems?
- Have we threat-modeled these integrations, and determined that we can mitigate or control factors to reduce the risks?
These, and many other similar analyses, are among the steps we recommend to our customers. We follow our own advice.
About This Document
As global society grapples with how, when, and why it uses AI software, we recognize that the first step for companies is to be transparent about what we do. We created this document to begin a conversation with customers and prospects so that they understand how we view these issues.
We also recognize that AI writing detection tools are both common, and often wrong, so we are writing this to set forth our commitment about the conditions under which we use AI.
One thing is clear to us: our commitment to conduct time-bounded engagements that have a pre-defined start and finish is made easier through the use of AI software. We find that using it to perform drudgery, to process and sort large quantities of unstructured data, and to perform tasks like spelling, grammar, and succinctness copy edits, as well as providing help as a coding assistant, all help us keep our promise of an average engagement of 45 days and a maximum engagement of 90 days. This means that we never allow AI to do our thinking for us, nor to create analytical conclusions or insights. Those are strictly in the realm of our subject matter experts.
Customer-Facing Reports
We use AI tools to assist in the creation of customer reports, and we feel it is important to set forth precisely what we do and do not do in this area. Plainly stated, our core intellectual property derives from the more than 130 years of collective expertise of our company’s partner-stockholders, and no insights or conclusions or advice is created by an AI tool.
Our flagship customer-facing CLARITY report (and the internal-only DISCOVER report that is its precursor) uses AI in the following manner:
All the questions that we ask customers in interviews, all the observations made by our partners and our associates in conducting those interviews, and derivative questions asked dynamically in interviews due to unexpected answers or newly discovered venues of investigation are generated by humans, using their understanding of the stated business challenges faced by our customers. We use an AI tool to transcribe the audio recordings of these customer interviews, and these transcripts are checked by our human collections and intake team for transcription accuracy.
All corporate documents provided by customers are handled as sensitive data (“Restricted” per the EPSD classification system, meaning that its distribution is tightly controlled, to a named distribution list, encrypted end-to-end), and human reviewed. A redacted copy of key passages and sections, containing only code names for companies, generic terms for departments, and no individual names, is fed into Claude for pattern matching of unstructured data. We have contractual attestation from Anthropic that this data, even though redacted, is not used for training or distributed to any third parties.
After these interviews and documents have been reviewed and analyzed, our intake team ensures that the correct EPSD partner and/or associate is assigned the areas specific to their personal subject matter expertise. In some cases, more than one partner will be assigned to an issue, because of overlapping concerns.
The human partners and associates conduct their analytical work “manually,” meaning they consider the problem at hand, and the context gained from the extensive interview process, and create an enumeration of problems which internally we refer to as “symptoms.” Then, the engagement lead studies these symptoms and conducts a diagnosis that can uniquely explain the presence of any given symptom. This is highly structured and iterative, and continues to ask, “If this diagnosis (the proposed root cause) did not exist, could this symptom exist? If the answer is yes, then we continue the search for root cause.
EPSD maintains a symptom library and has mapped hundreds of symptoms of dysfunction to dozens of categories of patterns we have seen in our careers. Our analysis continues until we have established a demonstrable one-to-one relationship between a given symptom and its root cause.
After these root cause-to-symptom matches have been established, we begin the report writing process. We use Claude to extract information in a highly structured format that gives us the data we need. The output is then reviewed by our report team, and materially and substantially re-written. That’s because, while we have found that Claude and other AI tools are able to synthesize large quantities of text far more quickly than a human, the quality of the writing they produce (from basic grammar and style, to logical leaps, to false connections) is universally sub-par.
Our report writing is designed to be clear to the non-technical executive reader in plain, straightforward, and logical business English, written without jargon wherever possible and explaining any jargon used immediately, so the reader can concentrate on the message, not the medium.
Once the rewrite has been completed, the newly completed second draft is circulated to the engagement team for comment, additions, corrections, and clarifications. A partner not on the engagement team is then given the opportunity to review the report, to determine whether someone without the context of those involved will be able to glean the meaning of the findings and the recommendations.
After this collaborative human review, another human copy edit is made, followed by an automated AI copy edit for succinctness, style, grammar, spelling, intra-document consistency, and clarity. This output is then comprehensively reviewed by the report team.
Only after this last human stage has been completed is the report placed into a Zero Trust-protected web sub-site. During a review meeting with the engagement lead and engagement consultants, this report is presented to the client, with key takeaways, recommendations, and action items. The online report is then shared with the client.
This policy was hand-written by a partner in a standard text editor, then collaboratively edited, clarified, and copy edited, and then we followed the procedure described in the two preceding paragraphs to produce the final copy you are reading now.
Internal Software Development
For our internal application development, our process is highly structured and involves software engineers, architects, and information security professionals conducting a formal and complete design of both the application to be built and the architecture we believe is the best way to build it. We maintain exacting standards of software functionality and information security testing, assuring that these tests are deterministic in nature.
Our approach is akin to those used for software development at the most successful companies: Agile techniques, small batch sizes, a robust continuous integration pipeline testing suite, and protected branches to ensure segregation of duties, among other things.
We create architectural decision record and decision record logs. Our changelog and commit files are richly detailed so that the genesis of our thinking is clear. There are breadcrumbs to follow to gain understanding of the problems faced, the solutions considered, and those rejected and accepted. One may also see evolution of our thinking over time, as these logs are append-only. Once the designs have been sketched out, our developers use the Claude Code AI assistant to help us formalize the documentation and ensure that it is comprehensive, internally consistent, and well-formed.
No software developed either by an AI coding assistant, or by hand, or a hybrid, is permitted to be pushed to a branch before it has passed the local test suite that mirrors exactly our CI testing suite. These tests include unit tests, smoke tests, regression tests, integration tests, and a range of information security and dependency checks from tools such as osv-scanner, Semgrep (with the Trail of Bits ruleset), and a dependency audit.