NB: Experimenting with different ways of structuring the titles for these

“Nothing here… what do you mean there’s nothing here?” Bob replied, striding toward Mikhail’s workstation. “Just what it sounds like. I opened the logfile from midnight last night and… nothing! Not even your typical network chatter.” “Let me see.” A worried look crosses Bob’s face as he almost jogs over to the computer. “Huh… you’re right. Not even ARP queries on the capture log. As far as I now there wasn’t any downtime last night until Scott called in. This is… not good.” Mikhail closes the file, taking a few screenshots of it just in case. He then opens the next entry in the folder. Again, nothing. Surprise turns to concern on his face, the unasked question hanging in the air between the two. Leaning in, Bob stops Mikhail from opening the next one in the series. “Hey, Mikhail. See anything with those last modified times?” Mikhail follows the technician’s finger. It seemed to be from a few minutes before. More interestingly, the time given was not uniform between the files. Someone or something had started modifying the entries in sequence, much the way that Mikhail would have looked at them. He scribbles a note on his pad of paper, documenting what he was seeing before his eyes. On a hunch he decides to reload the directory he was browsing and open several other Explorer windows, enabling him to look at each set of the log files from the affected time directly and simultaneously. Sure enough, as he had suspected the files’ last modified times were updating before his eyes except for one far down the listing - it still showed a time that matched when it was supposed to have been written. His hand trembling on the mouse, Mikhail double clicks on the file. ‘File is currently being edited by user Megasys\Scott.Mitchell. Would you like to open it in Read-Only mode?’ Mikhail and Bob share a look. “There has to be some explanation, right?” Bob asks after a few moments of silence. “Yep. There are a few different ways this could happen. Could be someone has stolen his password and we have an insider threat. Could be an account set up by our hacker made to look like Scott’s. Or…” Mikhail’s voice trails off, letting the third possibility go unsaid. “How can we tell?” “From here, we can’t. Need to be the domain admin or have the password to find out if that’s actually his. Alternatively, we could go ask Scott himself, right now.”

Bob gives Mikhail a look. He was not a confrontational type of person, doubly so when the person he would be confronting had the ability to fire him on a whim. “I know one of the system administrators fairly well, and besides, she owes me a favor. Maybe she can help us out.” Mikhail rolls his eyes, resisting the urge to cover his face with his palm. “Yes, go do that. I’ll stay here and pretend I didn’t hear you say that. Besides, there is another angle to the network traffic I can take a look at. These kinds of things leave one hell of a paper trail.”

Bob scurries off, happy to be more directly of use. Meanwhile, Mikhail opens a command prompt, thanking whatever IT gods might exist that Microsoft had finally, after more than a decade’s anticipation, fully integrated the bash shell into Windows with their Server 2028 R2 product, and that Megasys happened to be using that platform. That fact allowed Mikhail to use much of the same workflow that he was used to in his good old Linux environment resident on the laptop he had tried unsuccessfully to get onto the Megasys network. Fishing into his bag, he pulls out a USB stick and plugs it into his workstation. Such an action would raise eyebrows if the operations center was paying attention, but Mikhail figured that there were much more important things going on that he could get away with it. Besides, he wasn’t in Seattle all expenses paid on top of his (very generous) salary just to sit around on his hands, now was he? Loading up a network scanner, Mikhail finally takes a moment to check whether the network he could see matched the diagram he had been given. While that ran, he would attempt to see if anything had changed on the server he was connected to other than the changing log files. He also started up a traffic analyzer in case anything suspicious showed up while he worked. Such a step was ethically questionable, but from all he had understood there was nothing particularly stopping him from doing it. Nothing in any rules of engagement that he had been communicated, and nothing really stated verbally other than ‘do what you need to do’. Typically he would have known these things before even sitting down at the workstation, but so far nothing about this engagement was proving to be typical in the least. Most importantly, with each passing minute he felt less and less sure of himself; for a first solo assignment, Mikhail had been thrown into the fire. All he had to do was emerge on the other side with some kind of answer that would please the client.

He had been promised the access that he would need for this job, and Mikhail saw that he still had a while to go on the network scan he was undertaking. The type of scan he had chosen would take the better part of an hour even on small networks - checking even the C-suite’s subnet would take far longer, and he was getting as much as he could. He had time, in other words, to find out how much access he had. Mikhail picks a server on the list given to him by the CISO that was labeled db01-us-sfo. The primary database for their San Francisco network location. That would be a good place to start looking for any additional signs of intrusion; any attacker would want to get at the juicy information stored in database schema, whether it be customer details, financial records, or some important intellectual property. As promised, he is able to connect to the server. Nothing seems too out of place at first, so Mikhail decides to have a look at the logs; with luck, whoever or whatever had been altering them on the machine he had looked at before had not gotten to this one yet. Unlike the system he and Bob had tried to look at before, this one was apparently running the older style of logging - the one Windows admins have been used to for decades. Mikhail loads up the Event Viewer, and is reminded almost instantly why most systems administrators had switched to the newer logging format when he was still in school. A typical desktop computer could generate almost a million such records every year, and for servers that were used as heavily as the typical database server that he was on, that number was vastly eclipsed. So much so, in fact, that it had been deemed a best practice to clean out those logs on a regular basis - once they had been reviewed, of course. Thus, Mikhail lets out an audible groan when the Event Viewer loads. Either nobody at Megasys had thought to implement that particular recommendation, or in fact nobody was looking at these local system logs. The words of Scott Mitchell again echo in his head: at Megasys, best practice recommendations were gospel itself. Fortunately, an offline copy of the current state of the art in a searchable form was part of the standard issue his company sent him out with. With storage prices at almost inconsequential levels for a laptop, storing a large amount of plain text was not even worth thinking about, and furthermore it was not unheard of for Coyote Security consultants to go into a situation where the network itself was untrustworthy. As luck had it, those scenarios were increasingly where cellular service was spotty at best. Having a local copy just made sense.

Mikhail starts filtering the event logs. One benefit of the old style of logging was that the records were organized and sorted by severity as a default. It could be done with the new format, but most of that filtering was done by scanning the file of the log itself for a specific marker. With the older style, that flag was right in the title. Mikhail waits for the sorting to work itself out - only records from the last day were really meaningful to him at this moment. Though an attacker could remain inside a system for even months at a time before being discovered, right now Mikhail was just interested in pulling together some sort of picture of what had gone missing from the data loss and intrusion detection systems; that is to say that while actions with the database would be interesting later in the investigation perhaps, right now Mikhail wanted to find out if any clues had been left behind by whoever was responsible for wiping out the logs on the aggregating server he had tried to check before. It was a slim shot given the database servers would necessarily not have as much visibility on the network as the devices configured to look specifically for that sort of issue, but being that it was in itself often a target there was hope that maybe the person responsible had left something behind on this server too. Unfortunately for Mikhail, log files do not generally show “hey, I was hacked at 12:30 AM last night!” as something you can search for. It is quite literally the example of finding a needle in a haystack.

Mikhail happened to be fairly good at searching through haystacks. With as much time as he spent playing with technology both on the company’s and his own time, he had quickly become conversant in what most common systems were trying to tell him and when something might look out of place. A real administrator specializing in the given system would outclass him every time of course; it is simply not possible to be a master of everything. Mikhail’s specialty was in spotting patterns, finding the things that might seem innocuous, but are actually a sign of something wrong. This was typically a job for one of the many scripts he had written - instead of manually sorting through every log file looking for trends, he had written a fairly simplistic program that could handle much of that tedious work for him. Specifically, it looked for patterns in the logs - typical times of access, scheduled tasks, whatever constituted the rhythm of life for that particular server. Once Mikhail knew what the “normal” looked like, he could easily find things outside that pattern. Exceptions were not necessarily proof of malicious activity, but they were often good places to start. Anything to help narrow it down would be useful. Unfortunately for Mikhail, this server was by definition mission-critical for Megasys, and so he did not dare to run his script on it directly. In theory it should not cause any problems, but that was nice to say in theory, not necessarily when your boss is asking you why you caused a million dollar loss when the server went down. Stranger things had happened, and Mikhail knew some former coworkers who had been encouraged to move on for less than that. All that meant was that he had to be extra careful in this case. The server had some serious horsepower behind it, but strange things can happen. Looking at the event logs the “native” way was the only advisable course of action. If Mikhail brought down the server he would be in deep trouble… but if Microsoft’s code brought it down, then that would be an issue between Megasys and the company a few towns to the east. It was a pain in the ass to do it that way, but it was safe.

Mikhail quickly discovers that this server is rarely connected to directly, and always from the same set of Internet Protocol addresses. Whoever Megasys had employed as database administrators had managed to set it up so that only a minimum level of direct interaction was required; most routine tasks were handled by programs running on the server itself. Mikhail notes down the addresses that he sees most frequently as he scrolls through days of logs at a time. 10.30.127.2, 10.30.174.4, 10.28.102.37… Even now, rumors of the death of IPv4 had been greatly exaggerated and it kept persisting, a fact Mikhail was thankful for. Comparing IPv6 addresses by hand was at best annoying, and at worst something that would ruin his whole day. Then, as he scrolls back toward the present he finds something promising. A connection at 12:05 AM the previous night, matching with a disconnect thirty minutes later. Sticking out like a sore thumb is the IP address in question: 10.125.53.232. Something else, somewhere on the network had connected to the database. From the logs, Mikhail could not tell specifically what it had done other than a few specifics about the connection itself. He leans in and finds that a second set of connections had been established between the two he had highlighted. Except, from what Mikhail could tell from the logs, this set of connections was outgoing from the server he was connected to.

At that moment, Bob comes running back in, clearly out of breath and sweating from having run from across the building. He is followed a minute later by an older woman with dark hair in a shoulder-length ponytail and the sort of glasses that had been all the rage ten years prior. Unlike most of the rest of the Megasys personnel he had seen thus far, Mikhail immediately noted that this woman was wearing jeans and had an overall more casual look to her. “Mikhail, I’d like to introduce you to our senior Windows administrator, Mayra Khrishnan. She has agreed to help us out!” The look on Mayra’s face told Mikhail all he needed to know about how she had been convinced to stop what she had been doing.

“Hello, Mayra. I have been trying to get an idea of what happened last night, and thought that maybe the assistance of someone who knows the environment better than I could would be helpful.” Mikhail looks up as he finishes scribbling a note in his journal. “If we could just get this over with, that would be most excellent.” She says, flatly. “I do not have all day, like apparently some people around here, so let’s get right to it.” “Of course, I am sorry to have bothered you, however I noticed that logs on the aggregation server were being overwritten faster than I could load into them.” She scowls. This whole sequence of events had been far more than an inconvenience to her. “Overwritten? What do you mean, overwritten?” Mikhail explains how he had tried to look from the time of the incident forward, only to find that the log files were blank. Mayra stood there, emotionless, until he related the discovery of the user account that had apparently locked the file he had tried to read later. “You tell me that my boss’s boss’s boss was altering logs? That is a pretty big accusation to be making, don’t you think?”

Mikhail continues onward, showing her the screenshot he had taken. “With respect, Mayra, this appears to look like it. That is where I could use your help. Is this actually Scott’s account, or is it one that is made to make us think it is?” Mayra blinks. “Such a thing would be shown in the Active Directory user ID, if it were possible. I have yet to see evidence that it is, though. If anything, Microsoft was supposed to have fixed that possibility with the latest refresh. Except…” Her voice trails off. “I doubt I will find anything, but I will check on it for you. Goddess knows I spend too much time on Reddit as it is.” She rolls her eyes, giving a knowing look in Bob’s direction before running off.

“Well, that was… something.” Bob finally speaks up once Mayra is out of earshot. “That it was. She could use a cup of coffee, don’t you think?” Bob nods. “Anyway, while you were gone I think I found something.” “Oh?” “Yeah, take a look at this.” Mikhail gestures to the specific events he had highlighted. “Right around the time we are interested in, this connection opens up from the database server.” “And thirty seconds later the alarm went off.” Bob finishes Mikhail’s thought and is met with a nod. “Does that mean…” “Yeah, I think whatever is on the other end of our connection might be the suspect.”