The Mayhem for API Difference - A ZAP - Mayhem for API Scan Comparison

Sheldon Warkentin
September 7, 2022
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

With the rapid development of modern web APIs, developers must balance quality, reliability and security with time to market. When approaching testing, developers will aim for covering the “happy path”, that is, testing that the API behaves correctly under a strict set of conditions. While happy path testing provides some degree of confidence, one can expect any number of issues once the API is exposed to consumers.

API developers can try to find these issues early with negative testing, but without sufficient planning, negative testing is often ignored. This means that there are any number of unknown issues of unknown severity hidden in every API that may lead to benign errors, or at worse, serious security holes.

There are many tools that can be used to to fill in this gap, by “shifting left” the testing of APIs, using inputs that are meant to elicit insecure or unexpected behavior. One of these tools is ForAllSecure’s Mayhem for API.

Mayhem for API automates testing REST APIs by bringing the full might of fuzzing methodology to API testing. With the guidance of an API specification, Mayhem for API provides accurate and informative test coverage tailored to any REST API.

Another tool in this space is OWASP ZAP. ZAP is an open-source web application security scanner that can be used by both those new to application security as well as professional penetration testers. ZAP API scan is a script packaged with ZAP Docker images tuned for performing active scans against APIs. It is tuned to APIs, so it doesn’t bother looking for things like XSS

Both tools share the same goal:  Autonomously scan APIs to find security and implementation issues

In order to know what tool is right for you, we have put together some examples to see how both work with an example vulnerable REST API. Let’s begin! 

Prerequisites

  • Docker (with docker compose configured)
  • A Mayhem for API account. Sign up for Free! 
  • Mayhem for API cli, mapi, installed

The Target API

We have chosen VAmPI as our Target API.

VAmPI is a vulnerable API made with Flask and it includes vulnerabilities from the OWASP top 10 vulnerabilities for APIs.

It was created by GitHub user erev0s as a vulnerable API to evaluate the efficiency of tools used to detect security issues in APIs

Clone the VAmPI source from GitHub: https://github.com/erev0s/VAmPI

git clone https://github.com/erev0s/VAmPI.git 

Build a new image:

docker build -t vampi_docker:latest -f VAmPI/Dockerfile VAmPI 

Start the API using Docker Compose:

docker compose -f VAmPI/docker-compose.yaml up -d

Create a user in the application and get a bearer token:

curl --location --request POST "http://localhost:5002/users/v1/register" \
--header 'Content-Type: application/json' \
--data-raw '{
"email":"foo@example.com",
"username": "foo",
"password": "bar"
}'
curl --location --request POST "http://localhost:5002/users/v1/login" \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "foo",
"password": "bar"
}'

The response will contain the token you will use for other parts of the exercise

{
"auth_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NjIwNjkxNTIsImlhdCI6MTY2MjA2OTA5Miwic3ViIjoiZm9vIn0.7aB_94z7FmcGYNTaL67DW47Ht2WTBKlQ85eCbvmlBLM",
"message": "Successfully logged in.",
"status": "success"
}

Export the token to a variable to make it easier for the next calls.

export VAMPI_TOKEN="eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NjIwNjkxNTIsImlhdCI6MTY2MjA2OTA5Miwic3ViIjoiZm9vIn0.7aB_94z7FmcGYNTaL67DW47Ht2WTBKlQ85eCbvmlBLM"

Scan with Mayhem for API

Use the mapi CLI to start scanning the API

mapi run vampi auto ./VAmPI/openapi_specs/openapi3.yml \
--ignore-endpoint "^GET /createdb$" \
--header-auth "Authorization: token ${VAMPI_TOKEN}" \
--url "http://localhost:5002" \
--html mapi-report.html

Note some of the options used:

  •  --header-auth "Authorization: token eyJ0eX…
  • This uses the API token from the login. When scanning an API, you will be able to uncover more issues if you can successfully authenticate with it.
  • --ignore-endpoint "^GET /createdb$" \
  • The /createdb  endpoint should really only be called at most once per run to pre-populate the database. If it is called by the scanner, the results will be flaky as it recreates the database on every call.
  • --html mapi-report.html
  • This will create an html report with the results of your run.

At the end of the scan, your results will appear:

-Issues-
RULE_ID SUMMARY METHOD PATH
sql-injection GET /users/v1/{username}
internal-server-error AttributeError GET /books/v1
internal-server-error sqlite3.IntegrityError POST /books/v1
internal-server-error AttributeError GET /books/v1/{book_title}
internal-server-error sqlite3.OperationalError GET /users/v1/{username} 

Scan with ZAP - API Scan

Now we will try scanning the API with ZAP - API Scan. We will be calling ZAP - API Scan via the ZAP docker image.  A few more steps will be needed to get up and running. 

The target API is probably not in a great state after Mayhem for API has completed its run. Tear it down and start a fresh instance:

Start the API using Docker Compose:

docker compose -f VAmPI/docker-compose.yaml down 
docker compose -f VAmPI/docker-compose.yaml up -d

Don’t forget to create a user and get a new token!

curl --location --request POST "http://localhost:5002/users/v1/register" \
--header 'Content-Type: application/json' \
--data-raw '{
"email":"foo@example.com",
"username": "foo",
"password": "bar"
}'
curl --location --request POST "http://localhost:5002/users/v1/login" \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "foo",
"password": "bar"
}'

Set the token once more:

export VAMPI_TOKEN= &ltresult from previous request &gt

Let’s begin!

Create a working directory to mount and share with the ZAP Docker container:

mkdir zap_working_dir

Copy the VAmPI openAPI spec into the working directory and update it to point to the port exposed by Docker Compose:

cp ./VAmPI/openapi_specs/openapi3.yml zap_working_dir
sed -i '' 's/localhost:5000/localhost:5002/g' zap_working_dir/openapi3.yml 

Start running:

docker run --network=host -v $(pwd)/zap_working_dir:/zap/wrk \
-t owasp/zap2docker-weekly zap-api-scan.py \
-t "openapi3.yml" \
-f openapi -d \
-r zap-vampi.html \
-z "-config replacer.full_list\\(0\\).description=auth1
-config replacer.full_list\\(0\\).enabled=true
-config replacer.full_list\\(0\\).matchtype=REQ_HEADER
-config replacer.full_list\\(0\\).matchstr=Authorization
-config replacer.full_list\\(0\\).regex=false
-config replacer.full_list\\(0\\).replacement=token\\ ${VAMPI_TOKEN}
-config globalexcludeurl.url_list.url.regex='createdb$'"

Once the scan completes, you can find the results under ./zap_working_dir/zap-vampi.html.

Comparing the Tools

Results

We have collected the results from both tools and will dig into what was found by each. These results were obtained using the latest version of the software products as of September 1, 2022.


Severity

Issue Count


Description

ZAP

Mayhem for API

ZAP

Mayhem for API

High / Error

0

9

No high severity issues were found, including the SQL injection caught by Mayhem for API.

Mayhem for API discovered a SQL Injection and a number of internal server errors. The SQL Injection is a critical issue and should be resolved.


Internal Server Errors. While usually benign, as a result of an uncaught exception, these can lead to a poor experience for API consumers or indicate a deeper underlying issue.

Medium / Warning

2

9

Errors are triggered for missing CSP Header (which may be implemented at the load balancer rather than in the API) and Anti-CSRF token (which is more of an issue when cookies are involved – not with bearer token access).

Warnings range from issues with the API deviating from its OpenAPI specification, to timeouts on endpoints with performance that degrades after prolonged use. 

Low

5

N/A

Some low severity issues were found. These are mainly around  

web/application server is leaking version information via the "Server" HTTP response header and missing X-ContentType-Options Header.

Mayhem for API does not report issues below the Warning threshold.

Informational

2

N/A

This is mainly various 404 and 401 responses from the API

Execution

Results aren’t everything… both ZAP - API Scan and Mayhem for API are designed to be executed from the command line. Mayhem for API has been designed with an approachable command line user experience. Let’s compare how certain options, used in these examples, are expressed by each tool in the command line.  

Description

ZAP

Mayhem for API

Header Authentication

-z "-config replacer.full_list\\(0\\).description=auth1 -config replacer.full_list\\(0\\).enabled=true -config replacer.full_list\\(0\\).matchtype=REQ_HEADER -config replacer.full_list\\(0\\).matchstr=Authorization -config replacer.full_list\\(0\\).regex=false -config replacer.full_list\\(0\\).replacement=token\\ ${VAMPI_TOKEN}

The ‘-z’ option is used to pass configuration replacement values to the ZAP configuration. These arguments can be simplified by persisting a config file and passing it - at the cost of managing a config file. 

--header-auth "Authorization: token ${VAMPI_TOKEN}"

Header-based authentication is a first-class concept in Mayhem for API.

Specification Type

-f openapi

ZAP requires the specification type to be set up front

N/A


Mayhem for API will auto-detect your specification type, with support for swagger, OpenAPI, HAR and Postman Collections.

Specification Preprocessing

sed -i '' 's/localhost:5000/localhost:5002/g' zap_working_dir/openapi3.yml

ZAP requires that the openAPI spec server URL matches the target API. This means that the spec may need to be edited when running locally, against a different domain, or on a randomly assigned port. 

N/A


Mayhem for API will run against any URL provided by the ‘--url’ argument. If the specification is mostly valid, it will run!

Exclude/Include endpoints

-config globalexcludeurl.url_list.url.regex='createdb$'"


Config overrides can be used to exclude certain URLs from the specification with regular expressions. The specification itself can also be modified.

--ignore-endpoint "^GET /createdb$"

Mayhem for API makes it easy to configure exclude/include lists for endpoints and groups of endpoints (that use OpenAPI tags).

Conclusion

Both Mayhem for API and ZAP (with ZAP - API Scan) can be used to autonomously scan your REST APIs for quality and security issues. 

ZAP will help you to discover some inconsistencies in your API such as leaking server version information via HTTP Response Header, or returning Unexpected Content-Type. 

Mayhem for API will more consistently find high severity issues, such as SQL Injection - and helps you keep your specification honest with your implementation by identifying issues where the API is not behaving how it is defined in the spec. 

With command line options for setting header bearer token, endpoint exclusion/inclusion and no need to modify the OpenAPI spec URL prior to running - we believe that Mayhem for API is easier to get up and running against your APIs.

While we think that Mayhem for API is all you need for API scanning needs, we also understand the value provided by ZAP for organizations that are looking to cover all their bases. 

With Mayhem for API, you can include results from ZAP - API Scan in every run with the –zap command-line argument! 

mapi run … --zap

Or import results from your own ZAP runs:

 mapi run … --zap_import_json_results <path-to-zap-results.json>

Try it yourself!

Don’t just take our word for it, try it yourself!

Test this VAmPI and your own APIs by heading over to Mayhem for API to get started for free and start fuzzing!

Share this post

How about some Mayhem in your inbox?

Subscribe to our monthly newsletter for expert insights and news on DevSecOps topics, plus Mayhem tips and tutorials.

By subscribing, you're agreeing to our website terms and privacy policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Add Mayhem to Your DevSecOps for Free.

Get a full-featured 30 day free trial.

Complete API Security in 5 Minutes

Get started with Mayhem today for fast, comprehensive, API security. 

Get Mayhem

Maximize Code Coverage in Minutes

Mayhem is an award-winning AI that autonomously finds new exploitable bugs and improves your test suites.

Get Mayhem