DevSecOps: Integrating OWASP ZAP with GitLab, Calliope.pro functional tests and Slack notifications for reporting

This blog was originally posted to – https://devonblog.com/security/devsecops-integrating-owasp-zap-with-gitlab-calliope-pro-functional-tests-and-slack-notifications-for-reporting/

We were working with this particular software development team that was working with Mobile Applications, lots of APIs with a cross functional mix of API developers, Android and iPhone developers, QA personnel, architects, UI and UX folks and so on. We base our security consulting using the DevOn’s Continuous Software Security Maturity Model, and three months in to the engagement after some trainings, and initial security assessments it was time to plug in basic security scan as a part of the automation strategy.

Let’s get right to it. The development team that was working with GitLab for version control and Integration needs, with Calliope.pro as their test automation, and of course Slack for chatting (read as collaborating).

Here’s what we wanted to do:

1. Start OWASP ZAP

2. Use the matured API automated test suite that the team has developed via Calliope.pro

3. Let the automated tests proxy their traffic through OWASP ZAP

4. Wait for the functional automated tests to complete

5. Start active scan with OWASP ZAP (with the API-keys and session tokes that were proxied through OWASP ZAP)

6. Send the scan report to Slack

Well, there is many ways to do this, below is the way we chose to get up and running fast with minimal cost of setting and configuring all the nuts and bolts that work together.

Step 1:

Create a test job as “api-tests-proxy” that would run when triggered from Calliope.pro. This test job starts OWASP ZAP to listen on port 8090, then starts the functional automation suite whose traffic is proxied through 8090 to that OWASP ZAP could read the traffic, starts the active scan module of OWASP ZAP using the ZapScan.py file, then uploads the report to Slack

Step 2:

Write the ZapScan.py script to start the OWASP ZAP active scan, extract reports and publish message to Slack

Step 3:

Create and run the new test profile in Calliope.pro

That’s it. Now according to the schedule set in the test runner calliope.pro, the tests will run and reports will be published to the slack channel as intended

The above OWASP ZAP scan is not complete security scanning, nor is it fool proof security testing in any way. The described steps is just a small stepping stone in the entire DevSecOps transformation. The people that receive these reports need to know what’s a High, Medium, Low, or need to have the resources (for example access to a security professional or someone with the security knowledge) to decipher these reports and understand them.

N Different Strategies to Automate OWASP ZAP – OWASP APPSec BUCHAREST – Oct 13 2017

https://www.owasp.org/index.php/OWASP_Bucharest_AppSec_Conference_2017#tab=Conference_0101_talks

In this talk we will explore the many different ways of automating security testing with the OWASP Zed Attack Proxy and how it ties to an overall Software Security Initiative. Over the years, ZAP has made many advancements to its powerful APIs and introduced scripts to make security automation consumable for mortals. This talk is structured to demonstrate how ZAP’s API, and scripts could be integrated with Automated Testing frameworks beyond selenium, Continuous Integration and Continuous Delivery Pipelines beyond Jenkins, scanning authenticated parts of the application, options to manage the discovered vulnerabilities and so on with real world case studies and implementation challenges.
This is a demonstration oriented talk that explains OWASP ZAP automation strategies for Security Testing by example.

 

 

N Different Strategies to Automate OWASP ZAP – OWASP APPSec BUCHAREST – Oct 13 2017 from gmaran23

N Different Strategies to Automate OWASP ZAP – Cybersecurity WithTheBest – Oct 15 2017

http://cybersecurity.withthebest.com

In this talk we will explore the many different ways of automating security testing with the OWASP Zed Attack Proxy and how it ties to an overall Software Security Initiative. Over the years, ZAP has made many advancements to its powerful APIs and introduced scripts to make security automation consumable for mortals. This talk is structured to demonstrate how ZAP’s API, and scripts could be integrated with Automated Testing frameworks beyond selenium, Continuous Integration and Continuous Delivery Pipelines beyond Jenkins, scanning authenticated parts of the application, options to manage the discovered vulnerabilities and so on with real world case studies and implementation challenges.

 

N Different Strategies to Automate OWASP ZAP – Cybersecurity WithTheBest – Oct 15 2017 from gmaran23

 

 

OWASP ZAP: Global Exclude URL (Beta) – bug and fix

As you proxy your browser traffic through OWASP ZAP, chances are that you are annoyed by noise.  That is by default browsers these days make a lot of requests to update version, update cache, addons update and what not. It get’s really difficult to focus on the website at hand when you have other sites cluttering your Sites and History tab.

The Global Exclude URL functionality was supposed to work and it did work partially.

There was a minor bug and that was fixed.  A screen recording of the bug and the bug fix url below:

Global Exclude URL (beta) – after close and reopen does not pick up added regex for excluding URLs #3275

 

 

OWASP ZAP Development – Fixing the Can’t find bundle for base name lang.Messages error

 

I have been generating the API files for OWASP ZAP DOT NET API since the inception. There is the core zaproxy project that has the DotNetAPIGenerator.java class. And there is the extensions project, including the beta and alpha.

Now, when I tried to generate the ‘non-optional’, i.e., the core API files for .Net, everything would work fine, the API files would be generated as below.

 

OWASP ZAP is internationalized, so the source code comes with a bunch of resource bundles with supporting language files.

When you try to generate the API files for the extensions project, you get this wonderful error message.

Exception in thread "main" java.util.MissingResourceException: Can’t find bundle for base name lang.Messages, locale en at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1564) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1387) at java.util.ResourceBundle.getBundle(ResourceBundle.java:890) at org.zaproxy.zap.extension.api.AbstractAPIGenerator.(AbstractAPIGenerator.java:68) at org.zaproxy.zap.extension.api.JavaAPIGenerator.(JavaAPIGenerator.java:81) at org.zaproxy.zap.extension.ApiGenerator.main(ApiGenerator.java:73)

 

I have fixed this error message before when I was trying to generate the api files back in 2015. Running in debug mode and stepping through pointed out that the zaproxy core project had the resource files under a directory that was not available to the extensions project.

This error was gruesome.

In the end all I had to do was copy the contents of the workspaceowaspzap\zaproxy\src\lang directory to workspaceowaspzap\zap-extensions\bin\lang

That’s it. Do the same thing for the alpha, and beta extensions’ bin directory too.

 

Cheers. Try the OWASP ZAP DOT NET API available at nuget.org.

Downloading and Building OWASP ZAP source from Github using Eclipse IDE

 

Download this blog as PDF – https://renouncedthoughts.files.wordpress.com/2015/07/downloading-and-building-owasp-zap-source-from-github-using-eclipse-ide1.pdf 

 

This is a quick and dirty blog for those that are new to Eclipse IDE and want to try tweaking the OWASP Zed Attack Proxy’s code. I must say that that you might stumble upon this well written guide titled “Building OWASP ZAP Using Eclipse IDE for Java… Pen-Testers” here -  http://www.taddong.com/docs/Building_ZAP_with_Eclipse_v3.0.pdf . First time I was trying to build ZAP with Eclipse this guide was my complete reference. However, OWASP ZAP’s code was recently move to GitHub in the month of May-June 2015 rendering that guide obsolete and my OWASP ZAP Eclipse workspace – connected to google code SVN – a little defunct. Raul Siles, the author of the above guide would update it for changes with respect to the GitHub move.

Recently I was trying to download OWASP ZAP’s code from GitHub and build it because the existing code from SVN (google code) went obsolete. I am not an advanced Eclipse user or Java developer and I was a little lost trying to clone the new OWASP ZAP GitHub repo to my Eclipse. As I was trying, I took screenshots and ended up posted in this blog. Remember, this blog is not a step by step instruction, but it is a quick and dirty steps (5 major steps) to get OWASP ZAP’s code running in your Eclipse IDE.

Glimpse through the articles titled

1. Building OWASP ZAP Using Eclipse IDE for Java… Pen-Testers
2. Building ZAP (https://github.com/zaproxy/zaproxy/wiki/Building),
3. Downloading and Building OWASP ZAP source from Github using Eclipse IDE (this article)

and I am sure you’d get ZAP running on your Eclipse IDE.

1

Download Eclipse

…  from https://eclipse.org/downloads/. If you are confused which edition to download, pick the Eclipse IDE for Java Developers

 

When you open Eclipse for the first time choose the default workspace and proceed. If you’d like create a workspace such as workspaceowaspzap like I did. Refer to Raul Siles guide for workspace screenshots.

Make sure you have EGit plugin installed. If you are a prime time command liner with Git you may not need this plugin.

If you have downloaded Eclipse from Eclipse for Java Developers, then please ensure in the Eclipse Installation Details you have the below three components highlighted

1. Eclipse Git Team Provider
2. Java Implementation of Git
3. Mylyn Versions Connector: Git

At the time of this writing Eclipse IDE for Java Developers comes with all required plugins to work with Git ( and hence GitHub)

2

Add a Git Perspective

… to view Git Repositories and stuff..bla bla

Hit the Open Perspective button at the right top corner 

Choose Git at the Open Perspective Dialog

Hit OK to view the Git Repositories view.

Tip: From time to time you could hit the Java perspective to view the Java related tools and views, you could hit the Git perspective to view your Git Repositories.

 

If you look at the workspace that we choose when opening Eclipse, in Windows Explorer now it just has one folder named .metadata. Time to download the code from https://github.com/zaproxy 

 

3

Downloading the OWASP ZAP’s code

Choose File –> Import

Select Team –> Team Project Set. Hit Next.

In the Team Project Set Dialog, Input the Url –
https://raw.githubusercontent.com/zaproxy/zap-admin/master/ZAP-projectSet.psf 
and hit Finish.

Tip: Always refer to the recent project set Url available at https://github.com/zaproxy/zaproxy/wiki/Building

 

4

Wait for the ZAP projects to be downloaded and built

Watch the progress as the Git Repositories view would show projects as and when they are downloaded

Once all the ZAP projects are downloaded, your workspace the Git Repositories view should look like below. The approximate size of the workspace with all the ZAP coded summed up to 2.27 GB for me (on July 4 2015).

5

Run ZAP’s source and start playing (and contributing)

Switch to the Java perspective

In the Package Explorer, right click zaproxy and choose Run As –> Java Application

Eclipse would search for the Main types. In the Select Java Application dialog choose ZAP and hit OK

Witness the Console Logs

Start ZAPping

 

Tip: You can also start ZAP by hitting the play button in Eclipse

If you encounter any problems, try fixing it yourself first – spend a day or two , as a last resort – post at the ZAP Developer group here – https://groups.google.com/forum/#!forum/zaproxy-develop

OWASP ZAP Demonstration at OWASP Bangalore/Null meet on 22 Nov 2014

 

 

OWASP Zed Attack Proxy Demonstration – OWASP Bangalore Nov 22 2014 from gmaran23

 

 

http://null.co.in/event_sessions/171-owasp-zap-tool-demo

Abstract
The OWASP Zed Attack Proxy (ZAP) is an easy to use integrated penetration testing tool for finding vulnerabilities in web applications. Originally a fork of the Paros Proxy project, ZAP targets a wide range of software professionals right from a software developer to a penetration tester working on any platform that supports Java. Equipped with a myriad a features and support for custom addons, ZAP is fully documented in an easy to understand language.

We would see a demonstration of how to set up and how to use it all.
Speaker
Marudhamaran Gunasekaran

Timing
Starts at Saturday November 22 2014, 12:15 PM. The sessions runs for about 1 hour.

Beefing Up Security In ASP.NET Dot Net Bangalore 3rd meet up on May 16 2015

 

 

Beefing Up Security In ASP.NET Dot Net Bangalore 3rd meet up on May 16 2015 from gmaran23

 

Practical Security Testing For Developers Using OWASP ZAP at Dot Net Bangalore 3rd meet up on Feb 21 2015

 

 

Practical Security Testing for Developers using OWASP ZAP at Dot Net Bangalore 2nd meet up from gmaran23

 

Title	Practical Security Testing for Developers using OWASP ZAP
Abstract	Every time an application faces the world wide web, it inherently becomes vulnerable to attacks. The attackers could be script kiddies, joyriders, turning from hobbyists to downright hostile. The earlier in the development cycle you find the vulnerabilities, the better they are to fix and test. OWASP ZAP is a free and open source penetration testing tool for finding vulnerabilities in web applications; widely used by security professionals, it is also ideal for anyone new to web application security and includes features specifically aimed at developers. This session shows/demonstrates some attacks against web applications and how OWASP ZAP could be used to find those vulnerabilities, both manually and by automated builds.
Gist	See live attacks against web applications and how OWASP ZAP could be used to find those vulnerabilities, both manually and by automated builds.
Speaker	Marudhamaran Gunasekaran
Time & Venue	21 Feb 2015 @ Dot Net Bangalore 2nd meet up

OWASP ZAP : Workaround – Html Report from APIs daemon mode

 

Ok, may be the title is a bit misleading. The below post is going to show you a few workarounds that you can use to generate (or mimic) a html report when running OWASP ZAP automated in a headless mode.

Zed Attack Proxy (ZAP), as of version 2.3.1 stable release, provides options to generate an Xml report and an html report for the alerts found. We could get these two reports from the UI via the Report menu.

Use case:

One of the unique features of ZAP is the REST API that aids in automating scans. When running ZAP in daemon mode and pivoting ZAP from the REST APIs, there is a number of ways to harvest the alerts identified by ZAP.

Here’s a couple of them:

1. alerts based on id (http://localhost:7070/UI/core/view/alert/)
2. alerts for the baseurl (http://localhost:7070/UI/core/view/alerts/)
3. xmlreport (http://localhost:7070/UI/core/other/xmlreport/) [Same as clicking Report –> Generate Xml Report…]

Using the API and your favorite programming language, you could very well gather the alerts and decide what you want to do with it.

Problem:

One convenience is just pulling a html report out of the scan and mail the application owners or ourselves so they could be fixed. There is an enhancement request open in the projects page – https://code.google.com/p/zaproxy/issues/detail?id=1355

Until the feature is available in the next stable release, I decided to love with a couple of workarounds. There is a workaround in python, there is also a workaround in an exe format (written in C#). They are not perfect in terms of formatting, but they are close enough to the original html report generated through the ZAP UI.

Workarounds:

Download from here – https://github.com/gmaran23/HtmlReportThroughZapAPIs

 

Workaround #1:

A python script to inject an xslt stylesheet into the xml report file generated through the API. Use the sample function to insert the sample (ZapReport.xslt) into the xml file generated. Needless to say that the ZapReport.xslt and the xmlreportfile should be in the same location.

If needed to email the xml report, we can zip the xml report with the xslt file, so when unzipped the xslt file resides in the same location as the xml file.

When the xml file is opened in Firefox, or IE –> see the formatting in action! (Does not work in chrome though )

import io

def InsertXSLTSheetIntoXmlReport(xmlreportfile, xsltfile, xmlreportfileout):
    texttofind = '<?xml version="1.0" encoding="UTF-8"?>'
    texttoreplace = '<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="{0}" ?>'.format(xsltfile)
    with io.open(xmlreportfile, 'r', encoding="utf8") as f:
        xmlreport = f.read()
    xmlreportstyleinserted = xmlreport.replace(texttofind, texttoreplace)
    with io.open(xmlreportfileout, 'w', encoding="utf8") as f:
        f.write(xmlreportstyleinserted)


InsertXSLTSheetIntoXmlReport('SampleOWASPZAPReport.xml', 'ZapReport.xslt', 'SampleOWASPZAPReport-Mod.xml')

 

There is also another xslt template available from https://code.google.com/p/zaproxy/source/browse/trunk/src/xml/report.html.xsl

 

Workaround #2:

Use the command line program named XmlToHtmlWithXSLT.exe (requires .Net 4.0) as below to obtain a html report as output.

XmlToHtmlWithXSLT.exe SampleOWASPZAPReport.xml ZapReport.xslt converted.html

 

If you don’t have .Net 4.0, use the source below to recompile to any .Net version.

using System;
using System.IO;
using System.Text;
using System.Xml;
using System.Xml.Xsl;

namespace XmlToHtmlWithXSLT
{
    class Program
    {
        static void Main(string[] args)
        {
            string inputXmlFileName = args[0];
            string xsltfile = args[1];
            string outputHtmlFileName = args[2];

            XslCompiledTransform transform = LoadXsltTransform(xsltfile);

            StringWriter transformedToHtml = ApplyXsltTransform(inputXmlFileName, transform);

            WriteHtmlToFile(outputHtmlFileName, transformedToHtml);

            PrintStatus(outputHtmlFileName);
        }


        private static XslCompiledTransform LoadXsltTransform(string xsltfile)
        {
            XslCompiledTransform transform = new XslCompiledTransform();
            using (XmlReader reader = XmlReader.Create(xsltfile, new XmlReaderSettings() { DtdProcessing = DtdProcessing.Ignore }))
            {
                transform.Load(reader);
            }
            return transform;
        }

        private static StringWriter ApplyXsltTransform(string inputXmlFileName, XslCompiledTransform transform)
        {
            StringWriter transformedToHtml = new StringWriter();
            using (XmlReader reader = XmlReader.Create(inputXmlFileName, new XmlReaderSettings() { DtdProcessing = DtdProcessing.Ignore }))
            {
                transform.Transform(reader, null, transformedToHtml);
            }
            return transformedToHtml;
        }

        private static void WriteHtmlToFile(string outputHtmlFileName, StringWriter transformedToHtml)
        {
            using (StreamWriter outputFileStream = new StreamWriter(new FileStream(outputHtmlFileName, FileMode.Create)))
            {
                outputFileStream.Write(transformedToHtml.ToString());
            }
        }

        private static void PrintStatus(string outputHtmlFileName)
        {
            Console.WriteLine("Output Written to {0}", outputHtmlFileName);
        }
    }
}

 

Untill, the enhancement request (https://code.google.com/p/zaproxy/issues/detail?id=1355) is completed, these are some workarounds that I personally could live with, may be you can too, for your automating needs .

Worth looking at – https://www.owasp.org/index.php/GSoC2013_Ideas/OWASP_ZAP_Exploring_Advanced_reporting_using_BIRT