Listen on Apple Podcasts, Spotify and YouTube.
Welcome to AccelPro Audit, where we provide expert interviews and coaching to accelerate your professional development. Today we’re featuring a conversation with Mike Callino, Senior Director of Internal Audit at Braze.
We discuss the benefits of introducing new technologies into the audit process. Callino explains how implementing new methodologies, such as Agile, can increase successful collaboration across teams, as well as overall efficiency. He also foresees the increased use of AI in the auditing process, and claims it is already making data analysis more accurate and efficient.
“AI can streamline our audit tests, our audit fieldwork, and the like. And this is all within a controlled environment that we have within our GRC software. It's going to allow us to help write controls better, help process flow out better, write reports in a more timely manner, and help analyze big data sets.”
Listen on Apple Podcasts, Spotify and YouTube.
Interview References:
Mike Callino‘s LinkedIn profile.
1:54 | Sarbanes-Oxley Act of 2002, H.R.3763, (2002).
TRANSCRIPT
I. HOW TECHNOLOGY AND AI ARE STREAMLINING AUDIT
Alizah Salario, Host: You’re currently the Senior Director of Internal Audit at Braze, a digital-first company that strives to improve audience engagement through data-driven products. Can you give us an overview of your role in building up Braze’s audit function?
Mike Callino: Sure. I joined Braze in April 2021 after spending seven years at PwC. I arrived just before our IPO in November of the same year, inheriting a clean slate and the opportunity to shape the audit function. From PwC, I had a deep understanding of best practices. I worked a lot with technology companies.
Upon joining, my primary focus was on SOX compliance, given that we were about to undergo an IPO. I collaborated with a Big Four service provider. We conducted a thorough assessment covering processes, technologies, and controls, just really understanding a lay of the land.
This effort consumed about 80% of my time for the initial 18 months as we addressed a lot of remediation needs. As we enhanced our audit capabilities, I prioritized upscaling my team to align with a leading technology company like Braze. What this meant for me and the team was learning how to read code, learning how to write SQL, learning how to build data analytics and how to build automation.
We tried to make sure we were keeping pace with an evolving technology company where data is king. We needed to adapt and understand all of our data, how to analyze it and provide the most insights that we possibly could.
As the company matured and as we became a public company for a few years, I was able to build out the team. We now conduct operational audits; we build automations and partner with legal on enterprise risk management. It has definitely been a journey to stay data relevant, stay technology forward, and make sure my team is equipped to navigate the regulatory landscape.
AS: It sounds like there's a pretty steep learning curve, particularly when you are a technology forward company. What I'm curious about is how you develop the controls and processes around your products, some of which might not have a precedent in the marketplace. To put it in layman's terms, how do you manage emerging risks when you're charting new territory?
MC: We closely collaborate with cross-functional teams. These teams include our product teams, our engineering teams, and our data science teams. We really need to understand the technology stack used to support our products. This involves gaining a thorough understanding of both internally developed components—parts of our application that we fully developed and any third party services or integrations that speak to our product.
We conduct comprehensive risk assessments to identify potential threats and vulnerabilities unique to our platform. It's not a one size fits all. We need to really understand the various risks that there are to all the components of our platform.
This includes assessing risk related to access, change management, computer operations, and any dependencies on third parties service providers. As a software service company, you really can't do everything internally. You're always going to have an element of relying on third party service providers, so you really need to understand that third party risk element as well.
Next, we leverage industry best practices. Obviously you have best practices that are out there by ISO; you have COSO for internal controls. We want to make sure we're adapting to the most prominent and leading frameworks, so that we're designing our controls to be the strongest and most robust that could be tailored to our overall environment.
As it relates to implementing the controls, we want to make sure that we're gaining a comprehensive understanding of the various technology domains that may relate to them. We prioritize continuous monitoring and feedback loops, understanding that you may not get it all right the first time when you're designing or testing controls—when you're going through remediation.
We want to continuously have feedback loops to detect and address emerging risks that may arise as our technology evolves. This involves meeting regularly with management and conducting periodic risk assessments over our current controls and processes. Are they still relevant? Are they still effective? Or do things need to change? These are all things that are top of mind to me as we make sure we're maintaining a proactive stance and staying informed about industry trends and developments that could impact our overall environment.
AS: You mentioned collaborating with cross functional teams. I'm curious about what that really looks like with teams like IT, finance, and legal. Could you give us an example of how you work together and put those effective controls and risk assessments into practice?
MC: Collaborating with cross-functional teams is crucial for ensuring comprehensive risk coverage and effective processes and controls. At Braze, we adopt a multifaceted approach to achieve this goal. First, as I mentioned, we implemented SOX. SOX is your financial and technology controls, so there was definitely a huge partnership with the IT and finance teams to make sure that we had the right controls around our accounting processes, and also around the technology platforms that are used to support the processes.
This involves working closely with those stakeholders, establishing relationships, establishing credibility, and serving as a subject matter expert in the space of processes and control. To make sure that we have everything in place. As it relates to legal, we partner with legal to facilitate our enterprise risk management program.
Obviously from SOX, you only have one aspect of controls, which relate to internal controls of financial reporting, but as we matured as an organization, we wanted to have an enterprise risk management approach to make sure that we are considering risks across all areas: finance, operational, compliance, technology, and emerging risks.
We wanted to make sure we had more of a full holistic view on it, so we partner with legal to make sure that we are capturing our full risk universe. Our legal team partners with us to make sure our risk language is appropriate and everything aligns and makes sense from a risk profile standpoint.
We also emphasize continuous auditing and monitoring of high transaction areas. This means we internally utilize a lot of applications. We have a lot of data that we used to report on in our financials. We want to make sure that the integrity of our data is up to par, so we're constantly analyzing our data. In internal audit, we ingest the data, we write audit analytics against the data, and we provide insights to end users so that they can gain insights and understand, is there a process issue, is there a data integrity issue, and so forth.
What I'm trying to drive home here is that it's a constant collaboration and value provided process where we want to make sure that every function that we partner with understands the risks that are present within their area of expertise, and we wanna make sure they have the right mitigants in place to manage them.
AS: There's still this gap between what you do and what the leadership at your company might know or need to know. How do you communicate these concepts, particularly with people who not only aren't well versed in the audit universe, but they just simply might not be aware of data's capacity in 2024 and how you're using it to assess risk?
MC: Effectively communicating audit findings and recommendations, whether it be executive leadership, board members, senior members of the organization, especially regarding complex technical and emerging risks, is paramount in ensuring we as an organization are resilient and we're making informed decisions.
To dive a little bit deeper, we utilize straightforward language and avoid technical jargon to ensure that our message resonates with a diverse audience. There's a lot of audit jargon, a lot of complex terms, but it's really about knowing your audience. When you're speaking to executives and board members you want to distill complex technical concepts into easily understandable terms, focusing on the practical implications to the business. We also employ visual aids such as charts, graphs, and dashboards to present data and trends in a visually engaging format.
This enhances the audience's comprehension of the topic and facilitates quicker decision making by highlighting key takeaways and what they need to know. We provide context to our findings by illustrating the potential impact on the business objectives and strategic initiatives. We want to help executive leadership and board members understand the relevance and urgency of addressing the identified risks that we're presenting to them, whether it be in the audit committee or in ad hoc meetings.
AS: You work to develop clear and concise messaging. And I imagine that this is part of what helps you develop a culture of risk awareness and compliance. How else do you foster that culture, particularly among non technical staff?
MC: Among non-technical staff, what the internal function at Braze really emphasizes is education, engagement, accountability, and overall awareness. Starting with education, it's so important that everyone at the company who's involved in any compliance matter, understands what they're responsible for. If you're an accountant, you may be involved in a process. You may own a control. You need to have the education and background to understand that what you are doing is going to impact the financials.
I need to make sure, from an education standpoint, that I am providing learnings to you on an annual basis. We have a learning management platform. We have a compliance learning path. There are aspects that relate to SOXs and other regulations. We want to make sure that our employees are educated as it relates to compliance, financial reporting, and the like.
In terms of engagement, I want to make sure that any stakeholder is engaged and they don't view this as a ‘check the box’ activity. I want to make sure that if I'm spending time with you to improve your process, that you feel that way and you're not just doing something just because someone said you have to, so when we are sitting down and we're talking about process improvement or developing controls, I need to understand what keeps you up at night. What are the risks to you? And also layer that into what the regulations say we need to do.
By having that two-way engagement method of me telling you what the requirements are and you telling me what’s on your mind, that helps us build processes and controls that are relevant to what you're doing and allows you to embed them in your day to day.
Next, accountability. I want to make sure that control owners, process owners, anyone involved in compliance at Braze is held accountable and they understand the implications of not carrying out what they're expected to do. We have internal dashboards. We have control effectiveness rates. We analyze a lot of the data points. We want to make sure that we are keeping people up to date on their responsibilities. We also want to partner with you on remediation.
–
II: AGILE METHODOLOGY IN AUDIT
AS: Mike, you've given us an excellent overview of what you do at Braze at a high level, but I want to dig into the weeds of your work on a day-to-day basis. You are a proponent of Agile auditing. It's a topic that you've spoken about and presented about. And from what I understand, it's a set of guiding principles that prioritize customer satisfaction and collaborative approaches. Tell us a little bit about what it looks like in your work and why it's more effective and productive than, say, the waterfall approach, and traditional approaches to auditing.
MC: When I joined Braze in April of 2021, I became familiar with the term Agile as all teams applied an Agile methodology to manage their projects. I quickly had to do my homework and understand, what does Agile mean? After doing some research, I realized that I couldn't conduct audits in an elongated manner and needed to lean into Agile methodology.
Agile auditing, to me, focuses on breaking bigger bodies of work into sprints that allow you to complete audits or tasks on a more real-time basis. Technology companies are dynamic and I needed to deploy an auditing approach that could keep up as an example of how we deploy Agile app raises. Each year we compile our annual audit plan. We break each of these audits into projects. We use Asana for project management. Larger projects are time boxed into two week sprints that are allocated out to each part of the project. We then use a Kanban board, which is really just a visual display of project work so you could easily manage the lifecycle of your project work.
We take our full annual audit plan, we break them down into functional projects. We then, from the functional projects, break it down into two week sprints, and rather than doing a waterfall audit, where you would have a planning phase that may be six weeks, a field work phase that may be three months, a reporting phase that may be a month, and then remediation phase, that's a few months.
Those waterfall audits can take six to nine months. And by the end of the audit, what you have is your auditees or the people you're auditing are saying, “I didn't agree with that. I didn't tell you that because you've had a lot of these discussions in a phase that may have happened six months ago.”
When I deploy Agile into my auditing process, we break down our sprints. In one sprint, we do our planning, we do our field work, and we do our reporting. All in one. In a two week period, I'm going to do my planning, I'm going to conduct my field work, I'm going to align with my process owners, I'm going to communicate observations, and then I'm going to layer the results into my report.
Agile auditing allows me to conduct the audit more iteratively, more streamlined. It allows me to build my report over time and get concurrence from an auditee that they agree with the portion of the audit that we're completing.
Now, some listeners may say two weeks sounds very tight. It is not a one-size-fits-all two weeks of work for most of the audits that we do. Some sprints spill to three, four weeks, but it includes the same practice of time boxing work into smaller increments and making sure that you're covering all the risks in that given area, that you're completing the audit subcomponents and that you are finishing out timely so that by the end of the audit, you have small increments that build out your overall audit report.
This has been proven to be very valuable for my team because it allows us to understand the overall health of an individual audit. When we are prioritizing work, we'll have daily standups where we meet as a team and we analyze a Kanban board and we say, “What are you focused on? What's in progress? What can we put in the backlog? What came up in the business that we may need to work on now that I need to prioritize and can shift things around?”
As a manager, I don't need a micromanager to ask my team what they're doing. In each of their Asana Kanban boards, they have their weekly tasks that roll up to our overall audit plan and this allows me to understand the workflow, and allow us to conduct audits in a very efficient manner.
AS: There's tension between doing things fast, being efficient, and then also being really methodical and thorough, which are just essential traits for any auditor. How do you balance those two things in Agile auditing? And how do you think it helps you manage risk more effectively?
MC: It comes down to planning appropriately. When you plan for your full year's worth of work, we do spend quite a bit of time making sure that we are setting up our overall project plan with the larger projects and the smaller sprints that roll up to it in a way that's going to work for the team and not overload our work.
We want to make sure we're efficient. We want to make sure we're structuring the year in a way that's not going to be overwhelming, that could allow us to be successful. At the same time, there are fire drills that come up. There are ad hoc management requests that we need to deal with. With a tool like Asana and a Kanban board, I can visually move things into a backlog, or we can move things from in progress into a backlog, or move things around and get a full visibility to understand the health of our overall audit function.
This allows me to be efficient because I can move quickly, rather than having a weekly status meeting with my team and having a running Word Doc where I have no visibility into what we're actually working on. Even in an Excel project plan, I need to go to the project plan; I need to update things; I need to move things around. It's not a fluid environment, and working for a technology company, as an auditor, I need to be able to move quickly.
From an efficiency standpoint, it allows me to manage my team's workload, manage my overall audit plan, and understand where I have certain dependencies, as well as where we're at capacity. From a risk standpoint, it allows me the flexibility to field new requests in real time, and then look at my overall body of work and capacity and say, “We could do this or we can't do this,” rather than having an elongated process.
AS: How can AI help with repetitive or mundane tasks that could easily be automated?
MC: I think AI is here to stay. As it relates to how we utilize AI for auditing, we have a GRC software that actually has an AI co-pilot in it which we utilize for drafting up a control language, drafting up a remediation plan, helping us write audit reports, etc.
We utilize AI to streamline our audit tests, our audit fieldwork, and the like. This is all within a controlled environment that we have within our GRC software. It helps us write controls better, help process flow better, write reports in a more timely manner, and help analyze big data sets.
It can be used in a variety of ways. Obviously, as auditors, we're cautious. If I give the AI co-pilot something, what is it going to do with it? It's really about getting comfortable with your environment. Make sure you’re working in a closed environment with it.
AI’s not going to do everything well. It could do simple tasks very well: write a control for an AP accrual, write a control for a reconciliation. Simple tasks like that, it’s great at. However, if I wanted to write a full audit report, and I give it transcribed notes, it's not going to write the best audit report because that's not its subject matter expertise. But over time, as you train these models to understand prompts, to understand the nature of the work that you are doing, I think it's definitely going to disrupt everything that we do.
–
III: BETTER RESULTS AND MORE SUPPORT THROUGH NEW TECH
AS: It seems like we’re experiencing a massive change in not just auditing, but in many professions with the rise of AI and other technologies. How do you get others on board? Where do you see things going from here?
MC: I think as the industry is evolving continuously, everyone who is involved in auditing—whether it's internal auditing, external auditing—needs to embrace change and understand that the regulations and standards of today are not going to be the same as standards five years from now. Technology is going to disrupt everything that we do.
If we take an example of how we traditionally test controls, you usually test controls from a sampling standpoint. You may say, I have a population of 500. I want to test 25. That has been the way of the world for the past 20 years. If 25 tests clean, you would say a control is operating effectively.
Now with the power of technology, I could just analyze all 500 of those transactions and get more assurance than a reasonable assurance standard. I think the industry has evolved from a sampling standpoint to more of a full population analyzing coverage standpoint.
AI disrupts a lot of day-to-day tasks as it relates to drafting, control documentation, process flows, and analyzing big sets of data. I think automations and the use of GRC software and tooling is going to allow auditors to do their jobs more efficiently and effectively and is going to allow us to interconnect and provide more value to the business because everyone wants to move faster, and when you're using technology and tooling, it allows you to inherently move faster.
AS: I want to spend some time talking about your career path. Could you take us through some of the decisions that landed you where you are?
In April of 2021, I was at a point in my career where I was working a lot on technology and software at service companies. And the more I learned about their service offerings and the overall space, the more I was fascinated by how software is built from an infrastructure, from databases, from the underlying code, and the like.
At PwC, I had great experience working in all different industries, as you previously mentioned. But technology and software was the industry that really stayed near and dear to my heart. I had the fortunate opportunity to take a software service company public out of the New York metro area, in 2019. That whole experience excited me and made me want to take a company public on my own and go through the experience.
As a consultant either I was an outsourced consultant service provider, where I was the internal audit function and we reported to the audit committee, or I was a co-source provider, where they had maybe a small internal audit team and we were the arms and legs of the team, or sometimes I augmented the staff for a set project with set hours.
When consulting and working at PwC, you had your set of clients for as long as they needed you. And what I always knew I wanted to do was be part of a company and be part of the journey to build out an internal audit function from the ground up.
AS: I'm curious who you turn to when you're facing a challenge, or you just need a sounding board.
MC: I would say there's two sets of groups. One, my former directors that I used to work with who are now in the industry. They're actually my friends. My former directors, some of my former clients that I had at PwC, we are all now friends. If I have a question, if I need a call, if I need a sounding board to talk through navigating difficult situations, they'll always pick up the phone and talk through things with me.
And then secondly I utilize Slack. I am part of a chief audit executive network on Slack that has over a hundred chief audit executives across technology and other sectors. This allows me to reach out to that group if I have any questions on SOX, on new tooling or technology, whether another company is hiring and whether you may know anyone in your network.
This Slack community has been so great. We meet quarterly to discuss internal hot topics. We meet quarterly to discuss ERM hot topics, Enterprise Risk Management hot topics, and it's a very close group of Chief Audit Executives, where we share our ideas and we bounce things off of each other. I've been a part of it for about two years. I highly recommend it to other CIEs or leaders of IT audit or financial audit in the space; it’s definitely a really good forum to bounce ideas off of and get advice.
AS: It's pretty obvious after listening to you speak that this is an incredibly dynamic space and it's rapidly evolving. I know you don't have a crystal ball, but I'd like to know what you think is on the horizon for the audit profession and for your role in it.
MC: Auditing of the future will look much different than today. Tools and advanced technology such as analytics, AI, low code / no code automation will disrupt everything that we do. Auditors and management of companies will have greater risk coverage through deploying these sets of tools and making sure that their toolkit is readily available to manage the overall risk environment.
I look for ways to make my job more streamlined and efficient. I think about ways where I could get more risk coverage and think about how I could overall just keep learning and exploring, because I truly believe you need to stay relevant with technology in order to stay relevant in the audit industry.
Listen on Apple Podcasts, Spotify and YouTube.
This AccelPro audio transcript has been edited and organized for clarity. This interview was recorded on April 18, 2024.
AccelPro’s expert interviews and coaching accelerate your professional development. Our mission is to improve your day-to-day job performance and make your career goals achievable.
Send your comments and career questions to questions@joinaccelpro.com. You can also call us at 614-642-2235.
If your colleagues in any sector of the audit field might be interested, please let them know about AccelPro. As our community grows, it grows more useful for its members.
