If you are a pentester you probably never really think about programming. Instead you are testing what others have developed. However, every now and then a quick python or bash script is needed to exploit some stuff you have found, or automate a certain process you are using. 

Things become interesting when you are in a penetration testing company that has many strong penetration testers and everyone writes these scripts. Clearly each script solves a particular problem, either for the tester or the team. So how do we ensure, that:

• No time is wasted on writing the same scripts others have already written
• We keep the knowledge available when team members leave the business
• Ensure that any code is understandable for everyone
• Ensure that the knowledge in regards to the script is preserved for others to use.

The more scripts that are written, the more knowledge goes into code.
Code needs maintenance. Suddenly we find ourselves in the same situation as any other software development company. With a difference though, the life expectancy for a script is often just for a single project that lasts up to one month. Meaning, the code goes into hibernation, people forget about it or someone might remember it eight months later on another project. What you might end up with is an unmaintained self hosted gitlab instance with 200 projects, that no one knows anything about. What makes it even more chaotic is the usage of different languages that testers are comfortable with. Some might have a stronger background in Ruby, some really like using Golang. Others might write Typescript instead of Javascript. Don’t forget about Rust, all the C languages and maybe even some good old PHP. 

I want to share my knowledge about these issues and how we tackle them at JUMPSEC. We are fortunate enough to have a dedicated development team given we work on larger projects, such as our proprietary solution that is the backbone to our Continuous Attack Surface Management service. 

Understanding the basics

Before we tackle anything, we must understand the root cause for any problem. You might want to skip this section and jump straight into the solution areas below as this is non technical and very dry. 

At the very top, we should understand how important development is and understand the differences and challenges a penetration testing company has. 

In general, a developer gets paid to work on a product that solves a problem for a customer. They do not get paid to write code, as plenty of juniors might think. A penetration tester is getting paid to write a report that outlines the security issues a client has. In the background a developer must write code to make their product usable, while a security expert chooses to make use of programming to help with a certain problem (but might never need to). 

If we look at this from a very high level, it should be clear that a developer has much more experience handling code, compared to a general security researcher, simply due to the time they spend on programming. There are of course exceptions, where you have a team member that moved from development to cyber security in their professional career. But remember, you are only as strong as the weakest link. 

Which brings us directly to the people problem. It is already difficult to find new employees for a penetration testing company. The pool of experts is small and there is a worldwide shortage. Why would one make the pool even smaller, by requiring from a potential candidate that they know 10 different programming languages that might be used within the business? 

This brings us to some very important and specific requirements for our solution: 

• We must define what programming languages are used amongst the team
• Our solution cannot prohibit us from reducing the number of candidates 

Lets focus on the level of expertise and what we do with the weakest link and the most experienced developer on our pentest team. 

Generally, from my experience, penetration testers are not shy to play around with technology and are not afraid to break things. It is kinda part of the job description. So setting up a new environment for a project, like a C2 infrastructure is not really scarce. 

However, the approach between a developer, possibly with fullstack and devops experience, will differ from that of a security researcher. 

I believe in the year 2024 it is basic knowledge to set up stuff with docker and docker compose. However, I also think that it is not common to know your way around k8s, minicube and helm. Let alone I would never expect a pentester to be able to setup a full infrastructure as code project with Terraform, Ansible and Azure. Most certainly, not in an ad hoc situation for an adversarial simulation or similar pentest project. 

What if some members of the team are able to do that though? Well, the moment you start deploying projects in a very professional manner with lots of experience behind it, it becomes only manageable by the people that have that experience and exposure to the project. In other words, if your person leaves the business you start to have an unknown skeleton on your infrastructure, in some cases people might not even be able to login because of missing SSH keys. 

This brings us to the requirements:

• Technology used must be easy to learn and should not require lots of specialised expertise
• Must have the resources to onboard new team members quickly
• The goal must be that everyone in the business is able to jump on the maintenance call and get stuff solved
• Do not become dependent on a single person in the business

The solution

As we know from security and specifically ISO 27001 certifications, it’s all fine and great to have documents everywhere, but unless it really becomes the culture of the business, documents are useless. 

Unfortunately, we won’t be able to make it a culture without a document that clearly defines some processes and gives guidance to new members quickly. 

Quite frankly, a very well written document can speed up many different areas. 

Let’s call this document The Developer Handbook. 

If you have a look around the internet you will find many guides on how to write one. I would like to share with you our, real & currently in use, developer handbook. Furthermore, I will go into some details of some sections going forward. 

Let’s look into the sections that I believe are required for a successful document.

Format & Location

Documentation for development projects quickly gets outdated, let’s start by hosting it on Github and writing it in markdown, so any IDE / text editor can handle it. 

Versioning is also taken care of by working with git. After all, everyone that does coding should be familiar with some version control system. 

We will use pandoc to convert the markdown to a PDF version, and in fact an HTML version as well. 

Section 1 – Introduction

This section should cover the purpose of the document. Making it clear why people should read it and what the overall expectations are (both for the document and the users).

Section 2- Code of Conduct

As we are often building tools that make you think “what if this gets into the wrong hands”, we should remind ourselves why we do it. At the end of the day we are professional researchers that focus on security related subjects. The tools we write are there to make software behave in unintended ways, exploit vulnerabilities or make our life easier. 

Section 3 – Defining Programming Languages

In this section we should emphasise on our ability and set some limits. Sure it’s fun to explore new programming languages as a technical person. Frink, Rebol and Forth languages might look interesting during a weekend, but how many other people in the team could help out with those code bases? How quickly can issues be resolved by falling back to Stackoverflow and Google? Just because AI can help you with it, doesn’t mean it makes sense to use it in a business context. Keep those things to personal projects and find a common language everyone feels comfortable with. Usually that is: bash scripting, Python, sometimes Golang and lately more and more people are getting into Rust. 

Section 4 – Code Styles

This one is probably the most overlooked subject of all. However, it is incredibly useful & helpful to define a coding style for the entire team. It helps a lot, when everyone follows the same code style and no one needs to “adjust their way of reading”. 

Section 5 – Editors

Isn’t everyone on Sublime these days? Ah no, Atom looked so much nicer. What about Netbeans, PHPStorm, Eclipse (eh), vim, nano.  Well, and then there is VSCode. 

At the end of the day it doesn’t really matter which editor you use to write code in. What is more important is the set of features it can provide you to save time by automating tasks, such as formatting code to your set coding style. 

We went with VSCode. It is available in various “editions”, what matters for us is the extension ecosystem. We make heavy use of mypy, formatting and automated docstrings. solved within the editor and not just via pre-commit.

If you are using VSCode, this would also be a good place to have specific configurations shared, including default configurations for developer containers.

Section 6 – Version Control

First of all you want to specify what version control system you want to use. Whether it is git or svn. However, it is more important that this section can help as a “tutorial” on some processes. Sure you would expect everyone to know “git clone”, “git pull”, “git add” and “git commit”. But how about merging? Is this something you expect every consultant to know? What about the language of git? Staging & Stashing are probably the easiest ones. However, never assume that everyone is on the same page as you are. Specifically with development, things can go wrong or be done in many different ways. How do we delete files from the git history again so we don’t expose some secrets? Should we squash and if so when? How to do things and when to do them, without going over the entire git manual is probably a good starting point for this section.

Section 7 – Project Setup

The project setup section should cover a few things such as how a project should be structured both in terms of files, how dependencies are managed, what type of files are required. Best practices for public Github repositories, and generally how a project should be managed. 

Given that we are in a git environment, why not create an example project folder as well? 

Section 8 – Testing

There are many types of tests that can be written for any software project. Most likely performance tests are not the most important ones for pentesting software. However, this section should give some guidelines on what is expected and what testing suite should be used. 

It can also explain how to set up testing in a new project. How to use setUp and tearDown methods. 

Section 9 – CI & CD

Not every project needs a CI/CD process, however, whenever there is one it is imperative that certain actions are being executed. The section should talk about some limits in terms of resources to the company (not everyone has Github Enterprise with thousands of hours of worker times). Which workers should be used, what the authentication process looks like and how images need to be tagged.

Section 10 – Documentation

We are entering the inception phase with this one. Writing documentation and guidelines on how to document projects. However, it is a very important subject, as documentation often gets outdated very quickly. 

There are some solutions, such as self explaining code, building documentation automatically based on docstrings, comments etc. Rather than writing a handbook for each project, the time should be spent on the quality of the project. There might be a specific process that is involved to create documentation files automatically within the CI/CD as new versions are published. 

Section 11 – Security

Every programming language has some pitfalls. Specifically junior developers might fall for them. An obvious one might be the concatenation of strings rather than using f-strings. There are however many more and often some sort of approach or “library” should be established by developers within your company. Often code is reused, copy & pasted into new projects and simply adjusted. How do we check for security issues? What static code analysers should be used? Are there any requirements in the configuration of these? 

Section 12 – Performance

The performance section is quite an interesting one. Here, it is really up to you how deep you want to go  and how important performance considerations are. They certainly do matter, however, given that as a pentesting company your tools/products don’t serve 100 million end users it makes sense to show some approaches that might be better than introducing more advanced concepts such as multithreaded, multi core, async operations. Of course there are exceptions for certain tools. I suggest not to limit the section to pure technology based performance patterns, but also talk about general performance improvements within coding projects, such as best practices for code maintainability, documentation, dependencies, refactoring and so on.

Section 13 – Accessibility

We kept this section very short. We do not support any accessibility. All tools must be in English, no translations needed. Nor special support for operation systems, screen readers etc. Our tools are not for “real end users”, they are for the tech community and our team. 

However, if you do publish something that goes out to clients or end users, it might make sense to think about this section in more depth. 

Section 99 – Learning Resources

This is probably the most important section for new team members, junior team members and anyone that is starting on a project. The handbook should provide some valuable links and collections to tutorials and courses that the team can use to understand certain subjects. There is no harm to also link to your own blog posts. 

Give me a real example

Now that you have an idea of what sections there should be in a handbook, have a look at ours:

https://github.com/JumpsecLabs/Developer.Handbook

Final Words

A developer handbook is a constantly evolving document. The main purpose is to give guidance and contribute to the quality of development related work within a company. It should be adjusted every now and then, and it should never be considered a “finished product”. 

You will need to put the effort in yourself to create something that works for you and your team. This depends completely on the expertise of your team and what is important to your projects. There should be discussions with the team and every team member should be able to make changes to the document (remember section 99).

 

Bonus Section

Github automation

To give you a bit of technology in this blog post I figured it is best to quickly go over Github actions so that you can automatically generate the latest PDF version of the markdown files which you can then share with your team or store them somewhere for your ISO audit. 

We are using [pandoc](https://pandoc.org/) to automatically merge the different sections to a PDF. However, we cannot directly make it a PDF, we first must make a final HTML version. 

You can go over each step by reviewing the file: https://github.com/JumpsecLabs/Developer.Handbook/blob/main/.github/workflows/build-pdf.yml

There are two jobs involved: 

• Building the final PDF version
• Releasing it under the release tab on Github

Building a PDF

We cannot directly go from markdown to PDF with pandoc. Hence we must create an HTML version first. 

pandoc -t html --include-before-body=./version.md -s -o developer_handbook.html --toc --number-sections --wrap=none $(cat pandoc_order.txt)

This command takes all the files and in the same order as specified in the pandoc_order.txt file and converts them into a single standalone html file. 

Mind you, for the HTML version, we also have a css.md file, that allows us to make some nice colouring changes. 

Once we have the HTML version we run the converter to PDF

pandoc developer_handbook.html -s --pdf-engine=pdflatex -o developer_handbook.pdf -t pdf -f html

Have a look at the step “Install pandoc” in case you have issues here.

    - name: Install pandoc
      run: |
        sudo apt-get install texlive-latex-base texlive-latex-extra texlive-fonts-recommended pandoc -y

 

We are running this workflow on a base Ubuntu image and therefore need to install the latex package required for the pdf-engine. There are other engines as well, but we had some issues that pandoc did not find. In addition to the base installation some extras are necessary and for good measure we also put the fonts package in there. 

There is an texlive-fonts-extra package for fonts, however that increases the running time for the installing step immensely. 

There is an alternative way of doing this, and it is also documented on the official pandoc website: https://pandoc.org/installing.html#github-actions 

Releasing the PDF

We keep the approach very simple. We have a version.md file which we can update at our own will and bump the version manually. 

If you look closely at the “Extract Version” step in the release job you can see we extract it and later use it to tag our release upload on Github.

- name: Extract version
      id: extract_version
      run: |
        version=$(grep -oP 'Version: \K[0-9]+\.[0-9]+\.[0-9]+' docs/version.md)
        echo "::set-output name=version::$version"

Useful VSCode extensions

There are only two VSCode extensions I would recommend for this specific project type. 

Markdown All in One 

This one should be a no-brainer to everyone. However, did you know it can generate a table of contents for you? Open the command palette and run the command “Create Table of Contents”. 

Markdown as PDF

This one is a nice one if you want to create a quick PDF out of an MD file.