Computers, Programming, Technology, Music, Literature

Archive for February 2021

Install gedit and leafpad on kali linux

leave a comment »

apt-get update

apt-get install gedit

apt-get install leafpad

And please sudo as required.

image

image

image

Written by gmaran23

February 8, 2021 at 10:09 pm

Posted in kali

Tagged with , , ,

Set-AzWebApp: Case of The Unexplained 409 Conflict When Adding HostName

leave a comment »

tldr;

With Azure App Service, you could have the same Custom Domain assigned to multiple App Services at the same time. With one condition though: The same custom domain could not be assigned to two app services if there are in the same Deployment Unit <Scale Unit>.

When you call Set-AzWebApp with an updated Custom Domain Name (HostName), you expect the commandlet Set-AzWebApp to add the HostName to the desired web app. In case of a whatever error, you expect the Set-AzWebApp to throw an exception if the addition of the HostName failed.

Set-AzWebApp meticulously and miserably notifies you with a warning – not an error, a warning in a specific case that goes deep in to the Deployment Units (previously called Scale Units) and the App Service Plan you have chosen.

“

Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException:

Operation returned an invalid status code ‘Conflict’

”

Here’s a screenshot of Azure Release Pipelines silently ignoring the warning as it was supposed to (and Set-AzWebApp should have thrown an error in possible senses instead of a warning), and letting the further dependent tasks fail.

image

With advise from one of my colleagues, when looked at the WebApp to which we were trying to add the host name to, it revealed another detailed error.

"statusMessage": "{\"Code\":\"Conflict\",\"Message\":\"The host name redacted is already assigned to another Azure website: redacted.\",\"Target\":null,\"Details\":[{\"Message\":\"The host name redacted is already assigned to another Azure website: redacted.\"},{\"Code\":\"Conflict\"},{\"ErrorEntity\":{\"ExtendedCode\":\"54004\",\"MessageTemplate\":\"The host name {0} is already assigned to another Azure website: {1}.\",\"Parameters\":[\"redacted\",\"redacted\"],\"Code\":\"Conflict\",\"Message\":\"The host name redacted is already assigned to another Azure website: redacted.\"}}],\"Innererror\":null}",

image

With Azure App Service, you could have the same Custom Domain assigned to multiple App Services at the same time. With one condition though: The same custom domain could not be assigned to two app services if there are in the same Deployment Unit <Scale Unit>.

Here’s some additional documentation from msdn that details out the condition.

https://docs.microsoft.com/en-us/azure/app-service/manage-custom-dns-migrate-domain#migrate-domain-from-another-app

You can migrate an active custom domain in Azure, between subscriptions or within the same subscription. However, such a migration without downtime requires the source app and the target app are assigned the same custom domain at a certain time. Therefore, you need to make sure that the two apps are not deployed to the same deployment unit (internally known as a webspace). A domain name can be assigned to only one app in each deployment unit.

You can find the deployment unit for your app by looking at the domain name of the FTP/S URL <deployment-unit>.ftp.azurewebsites.windows.net. Check and make sure the deployment unit is different between the source app and the target app. The deployment unit of an app is determined by the App Service plan it’s in. It’s selected randomly by Azure when you create the plan and can’t be changed. Azure only makes sure two plans are in the same deployment unit when you create them in the same resource group and the same region, but it doesn’t have any logic to make sure plans are in different deployment units. The only way for you to create a plan in a different deployment unit is to keep creating a plan in a new resource group or region until you get a different deployment unit.

Written by gmaran23

February 5, 2021 at 7:09 pm

Posted in Azure, powershell

Tagged with , , , , ,

Powershell UnitTesting with Pester: Return a Different Mock Response For the Second or Nth Time

leave a comment »

Credits: https://github.com/pester/Pester/issues/574

Let’s say you want to update the ‘Custom Domain Name’ for an App Service in Azure through powershell. Because Azure commandlets and API’s could sometimes be flawed, after updating, you want to verify whether the ‘Custom Domain Name’ is added to the App Service.

One way to achieve that is as in the commandlet below:

1. Get-AzWebApp
2. Add the HostName
3. Set-AzWebApp
4. After setting, Get-AzWebApp
5. Verify if the HostName is added

Now, when mocking Get-AzWebApp, because we are calling Get-AzWebApp twice, we should be able to return a different response the second time Get-AzWebApp is called. There is a suggestion to use a $script level or a $global level variable called $mockCounter or $mockcounterwhatevernameforthecounteryouwanttogive here in the the Pester Issues.

Here’s a working example of an actual commandlet, and a unit test. 

function Add-RenTWebAppDomainName {
    <#
    .SYNOPSIS
    Adds a custom domain for an App Service.
    .DESCRIPTION
    Adds a custom domain for an App Service.
    .EXAMPLE
    Add-RenTWebAppDomainName -ResourceGroupName 'RenT-12' -WebAppName "rentapp" -Hostname 'rentapp.renounced.com'
    .PARAMETER ResourceGroupName
    The resource group in which the App Service resides.
    .PARAMETER WebAppName
    The name of the App Service to which a custom domain name needs to be added.
    .PARAMETER HostName
    The custom domain name that needs to be added to the App Service.
    #>

    [OutputType()]
    [Cmdletbinding()]
    param(
        [Parameter(Mandatory = $true)]
        [string]$ResourceGroupName,

        [Parameter(Mandatory = $true)]
        [string]$WebAppName,

        [Parameter(Mandatory = $true)]
        [string]$HostName
    )

    begin {
        Write-RenTLogStartCmdlet -CallerInvocation $MyInvocation
    }

    process {
        try {
            Write-Verbose ("Associating custom domain name '{0}' for the app {1}" -f $HostName, $WebAppName)

            $WebApp = Get-AzWebApp -Name $WebAppName -ResourceGroupName $ResourceGroupName
            $HostNames = $WebApp.HostNames
            $HostNames.Add($HostName)
            Set-AzWebApp -Name $WebAppName -ResourceGroupName $ResourceGroupName -HostNames $HostNames

            Write-Verbose ("Verifying if the custom domain name '{0}' for the app {1} is added" -f $HostName, $WebAppName)

            $WebApp = Get-AzWebApp -Name $WebAppName -ResourceGroupName $ResourceGroupName
            $HostNames = $WebApp.HostNames
            if($HostName -in $HostNames){
                Write-Verbose ("Successfully associated custom domain name '{0}' for the app {1}" -f $HostName, $WebAppName)
            }
            else{
                Write-Error ("Could not associate custom domain name '{0}' for the app {1}" -f $HostName, $WebAppName)
            }
        }
        catch {
            Write-Error $_.Exception.Message
        }
    }

    end {
        Write-RenTLogEndCmdlet -CallerInvocation $MyInvocation
    }
}

However, when you want to Unit Test the commandlet, you want to return a different response the second time Get-AzWebApp is called. Here’s a sample with 100% code coverage:

. (Join-Path $PSScriptRoot ../../private/logging/Write-RenTLogEndCmdlet.ps1)
. (Join-Path $PSScriptRoot ../../private/logging/Write-RenTLogStartCmdlet.ps1)
. (Join-Path $PSScriptRoot Add-RenTWebAppDomainName.ps1)

Describe "Add-RenTWebAppDomainName" {

    function Set-AzWebApp {[CmdletBinding(SupportsShouldProcess)] param($Name,$ResourceGroupName,$HostNames) if ($Force -or $PSCmdlet.ShouldProcess("ShouldProcess?")){} return}

    Context "A HostName is supplied for a WebAppName in a ResourceGroupName" {
        It "It adds the HostName (Custom Domain Name) to the WebApp." {
            $script:mockCounter = 0;
            function Get-AzWebApp {}
            Mock Get-AzWebApp {
                $returnValue = $null
                if ($script:mockCounter -eq 0){
                    $returnValue = [PSCustomObject]@{ Name = 'WebAppName'; HostNames = [System.Collections.Generic.List[System.Object]]@('HostName1')};
                }
                else{
                    $returnValue = [PSCustomObject]@{ Name = 'WebAppName'; HostNames = [System.Collections.Generic.List[System.Object]]@('HostName1','HostName2')};
                }
                $script:mockCounter++
                return $returnValue
            }
            Mock Set-AzWebApp { }

            Add-RenTWebAppDomainName -ResourceGroupName 'RG' `
                -WebAppName 'WebAppName' `
                -HostName 'HostName2' `
                -ErrorAction Stop

            Assert-MockCalled Get-AzWebApp -Times 2 -Exactly -Scope It
            Assert-MockCalled Set-AzWebApp -Times 1 -Exactly -Scope It
        }

        It "It runs with errors." {
            Mock Get-AzWebApp -MockWith { throw "Mocked error" }

            { Add-RenTWebAppDomainName -ResourceGroupName 'RG' -WebAppName 'WebAppname' -HostName 'HostName' -ErrorAction Stop } | Should Throw "Mocked error"

            Assert-MockCalled Get-AzWebApp -Times 1 -Exactly -Scope It
        }

        It "Fails to add the HostName for the WebApp"{
            $script:mockCounter = 0;
            function Get-AzWebApp {}
            Mock Get-AzWebApp {
                $returnValue = $null
                if ($script:mockCounter -eq 0){
                    $returnValue = [PSCustomObject]@{ Name = 'WebAppName'; HostNames = [System.Collections.Generic.List[System.Object]]@('HostName1')};
                }
                else{
                    $returnValue = [PSCustomObject]@{ Name = 'WebAppName'; HostNames = [System.Collections.Generic.List[System.Object]]@('HostName1')};
                }
                $script:mockCounter++
                return $returnValue
            }
            Mock Set-AzWebApp { }

            { Add-RenTWebAppDomainName -ResourceGroupName 'RG' `
                -WebAppName 'WebAppName' `
                -HostName 'HostName2' `
                -ErrorAction Stop } | Should Throw "Could not associate custom domain name 'HostName2' for the app WebAppName"

            Assert-MockCalled Get-AzWebApp -Times 2 -Exactly -Scope It
            Assert-MockCalled Set-AzWebApp -Times 1 -Exactly -Scope It
        }
    }
}

Written by gmaran23

February 5, 2021 at 5:11 pm

Posted in powershell, Testing

Tagged with , , ,

Changing the Default Postgres User Identity

leave a comment »

The default identify that ships with the package to run postgresql is postgres. Change it to postgresuser

sudo adduser postgresuser

sudo passwd postgresuser

 

/database/ is the mount for database files, so the default directory for postgres to be changed

Create a directory /pgsql/data under /database/

sudo mkdir /database/pgsql/data -p

Then change the ownership of /database/pgsql/data to the user postgresuser

sudo chown -R postgresuser:postgresuser /database/pgsql/data

Also change the ownership of the default install directory

sudo chown -R postgresuser:postgresuser /var/lib/pgsql

Delete the default postgres user

sudo userdel -r postgres

Look at the default system unit file for postgresql and extend the unit file configuration to override the default postgres user to postgresuser

cat /lib/systemd/system/postgresql.service

 

 

Extending systemd service unit files are extensively documented at  – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-managing_services_with_systemd-unit_files#sect-Managing_Services_with_systemd-Unit_File_Modify

Create a directory postgresql.service.d under /etc/systemd/system

sudo mkdir /etc/systemd/system/postgresql.service.d/

Create a file named overrideuser.conf

sudo touch /etc/systemd/system/postgresql.service.d/overrideuser.conf

Edit the overrideuser.conf to provide User and Group information as the newly created postgresuser.

sudo vi /etc/systemd/system/postgresql.service.d/overrideuser.conf

 

[Service]

User=postgresuser

Group=postgresuser

 

:wq!

 

The configuration files from configuration directories in #/etc/systemd/system/ take precedence over the default unit files in #/usr/lib/systemd/system/.

Change the ownership of the /var/run/postgresql where the postgres will create a socket when running

sudo chown -R postgresuser:postgresuser /var/run/postgresql

/var/run/postgresql directory will be erased on every reboot, hence the directory need to be created during reboot with the ownership of the newly created postgresuser.

sudo vi /usr/lib/tmpfiles.d/postgresql.conf

d /var/run/postgresql 0755 postgresuser postgresuser –

:wq!

Failing to correctly modify the ownership and configuration of the /var/run/postgressql might results in errors like below:

sudo systemctl status postgresql.service -l

# FATAL: could not create lock file

#"/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied

After the above changes, Enable postgresql.service to be run on system start, then reload the systemd daemon, start the postgresql.service, check the status to see if the service is running.

sudo systemctl enable postgresql.service

 

sudo systemctl daemon-reload

 

sudo systemctl restart postgresql.service

 

sudo systemctl status postgresql.service -l

 

Now postgresql.service should be running under the newly created user identify postgresuser.

Written by gmaran23

February 5, 2021 at 4:37 pm

Posted in devsecops, security

Tagged with ,

Securing postgresql authentication methods

leave a comment »

Review the authentication methods:

https://www.postgresql.org/docs/11/auth-methods.html

https://www.postgresql.org/docs/11/auth-password.html

 

postgresql by default users ident and trust for local networks and the default authentication methods are found at

sudo cat /var/lib/pgsql/data/pg_hba.conf

 

The password storage mechanism for postgresql users is configured at

sudo cat /var/lib/pgsql/data/postgresql.conf

 

Starting postgresql11, there is support for SHA-256 based passwords with the configuration scram-sha-256. So, change the default password hashing mechanism for postgresql users at postgresql.conf.

 

sudo vi /var/lib/pgsql/data/postgresql.conf

password_encryption = ‘scram-sha-256’

 

:wq!

 

Change the authentication settings in pg_hba.conf

Change peer to trust and ident to scram-sha-256

 

sudo vi /var/lib/pgsql/data/pg_hba.conf

 

 

 

# TYPE DATABASE        USER            ADDRESS                 METHOD

 

# "local" is for Unix domain socket connections only

local   all             all                                     trust

# IPv4 local connections:

host    all             all             127.0.0.1/32            scram-sha-256

# IPv6 local connections:

host    all             all             ::1/128                 scram-sha-256

# Allow replication connections from localhost, by a user with the

# replication privilege.

local  replication     all                                     peer

host   replication     all             127.0.0.1/32            ident

host   replication     all             ::1/128                 ident

 

Reload and start service after authentication change.

 

sudo systemctl status postgresql.service -l

sudo systemctl daemon-reload

sudo systemctl restart postgresql.service

sudo systemctl status postgresql.service -l

 

Verify authentication changes by logging in to psql and checking version.

su – postgresuser

 

psql –version

select * from pg_shadow;

List users, and change the password for the database user named postgresuser.

\du+

\password

After password change the ps_shadow table should show a password for the postgresuser starting with scram-sha-256.

select * from pg_shadow;

Exit from pgsql, and postgresuser.

Change the pg_hba.conf to enforce scram-sha-256 for local connections as well.

 

sudo vi /var/lib/pgsql/data/pg_hba.conf

Change trust to scram-sha-256

 

# TYPE DATABASE        USER            ADDRESS                 METHOD

 

# "local" is for Unix domain socket connections only

local   all             all                                     scram-sha-256

# IPv4 local connections:

host    all             all             127.0.0.1/32            scram-sha-256

# IPv6 local connections:

host    all             all             ::1/128                 scram-sha-256

# Allow replication connections from localhost, by a user with the

# replication privilege.

local  replication     all                                     scram-sha-256

host   replication     all             127.0.0.1/32            scram-sha-256

host   replication     all             ::1/128                 scram-sha-256

 

:wq!

Reload the daemon, and restart the postgresql.service.

sudo systemctl daemon-reload

sudo systemctl restart postgresql.service

sudo systemctl status postgresql.service -l

 

 

As a last change, login to postgres and delete the default user database ‘postgres’.

 

su – postgresuser

psql

 

Enter the newly created password for the database identify postgresuser.

 

List all databases

\l

drop database postgres;

Written by gmaran23

February 5, 2021 at 4:26 pm

Posted in continuous security, continuous security, devsecops, security

Tagged with ,

Azure Resources Migration between subscriptions – code snippets

leave a comment »

Credits: Vijayaragavan Inbaraj, and Mohammed Siyam

Before migration, review the supported resources at https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-support-resources

If the source and the destination subscription are in different tenants, then you’d want to create a ‘temp’ destination subscription, and change the ‘temp’ subscription’s tenant to the new tenant.

Steps:

  1. Check if quotas in the new subscription are equivalent to the quotas in the old subscription
  2. Check if the supported "Resource Providers" in the new subscription are in "Registered" state as the old subscription
  3. Gather a list of Resource Groups to be moved (filter by tags or names or a another filter)
  4. Gather a list of resources that do not support movement between subscriptions [Move operation support for resources]
  5. Talk to the team members, resource group owners, product owner and obtain their confirmation for the move
  6. Communicate possible downtime if any
  7. Create new empty resource groups with the same name and tags as in the old subscription
  8. Initiate the move command for one resource group at a time because it is safer to do it in case any errors occur
  9. After the resources in a resource group are moved, check for the number of resources in the moved Resource Group to see if all the resources are moved without fail
  10. Create a "Do-Not-Delete" Lock on the new resource groups that should not be deleted
  11. After verifying, delete the old resource groups
  12. Communicate

Below are some snippets that would assist in the above migration process.

Note:

1. Adjust the $whatif flags accordingly
2. Use the –Debug flag for Az commandlets to get more information on non-revealing error messages

############################################################################################################
#Snippet to check quotas
############################################################################################################


Set-AzContext -SubscriptionId 'oooooooo-oooo-llll-llll-dddddddddddd' #old-subscription
Get-AzNetworkUsage `
  -Location westeurope `
  | Where-Object {$_.CurrentValue -gt 0} `
  | Format-Table ResourceType, CurrentValue, Limit
 
  Get-AzStorageUsage `
  -Location westeurope `
  | Where-Object {$_.CurrentValue -gt 0}
 
  Get-AzVMUsage `
  -Location westeurope `
  | Where-Object {$_.CurrentValue -gt 0}

———– 

############################################################################################################
#Snippet to register missing resource providers in the new subscription
############################################################################################################


Set-AzContext -SubscriptionId 'oooooooo-oooo-llll-llll-dddddddddddd' #old-subscription
$ExistingResourceProvidersInOldSubscription = Get-AzResourceProvider


Set-AzContext -SubscriptionId 'nnnnnnnn-nnnn-eeee-eeee-wwwwwwwwwwww' #new-subscription
$ExistingResourceProvidersInNewSubscription = Get-AzResourceProvider


foreach($ExistingResourceProviderInOldSubscription in $ExistingResourceProvidersInOldSubscription) {
    if($ExistingResourceProviderInOldSubscription.RegistrationState -eq 'Registered'){
        $providerfound = $false;
        foreach($ExistingResourceProviderInNewSubscription in $ExistingResourceProvidersInNewSubscription){
                if($ExistingResourceProviderInNewSubscription.ProviderNamespace -eq $ExistingResourceProviderInOldSubscription.ProviderNamespace){
                    $providerfound = $true;
                    if($ExistingResourceProviderInNewSubscription.RegistrationState -eq 'Registered'){
                        Write-Host '------------------------------'
                        Write-Host $ExistingResourceProviderInOldSubscription.ProviderNamespace 'PRESENT, REGISTERED'
                        Write-Host '------------------------------'
                    }
                    else{
                        Write-Host $ExistingResourceProviderInOldSubscription.ProviderNamespace 'PRESENT, UNREGISTERED, Registering'
                        Register-AzResourceProvider -ProviderNamespace $ExistingResourceProviderInOldSubscription.ProviderNamespace -WhatIf:$WhatIf
                    }                        
                }
            }
        if(-Not($providerfound)){
            Write-Host $ExistingResourceProviderInOldSubscription.ProviderNamespace 'ABSENT, Registering'
            Register-AzResourceProvider -ProviderNamespace $ExistingResourceProviderInOldSubscription.ProviderNamespace -WhatIf:$WhatIf
        }
    }
}

———— 

############################################################################################################
#Snippet to gather resource groups to be moved
############################################################################################################

#A Class to hold information to be used for discussion later
class Resource{
    [string]$ResourceGroupName
    [string]$Name
    [string]$Moveable
    [string]$Owner
    [string]$Comments;

    Resource(
    [string]$ResourceGroupName,
    [string]$Name,
    [string]$Moveable,
    [string]$Owner,
    [string]$Comments
    ){
        $this.ResourceGroupName = $ResourceGroupName
        $this.Name = $Name
        $this.Moveable = $Moveable
        $this.Owner = $Owner
        $this.Comments = $Comments
    }
}


$ResultObject = @([Resource]::new('--', '--', '--', '--', '--'))


Set-AzContext -SubscriptionId 'oooooooo-oooo-llll-llll-dddddddddddd' #old-subscription


# https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-support-resources
$NonMovableResourceTypesWithinSameTenant = @('Microsoft.AAD/DomainServices', `
'Microsoft.ContainerService/managedClusters', `
'microsoft.insights/activityLogAlerts', `
'microsoft.insights/metricalerts', `
'microsoft.visualstudio/account', `
'Sendgrid.Email/accounts')


$StaticRG = Get-AzResourceGroup

<#
$StaticRG = Get-AzResourceGroup | 
Where-Object {$_.Tags -and ($_.Tags.GetEnumerator() | 
Where-Object {($_.Value -eq 'Development') -or ($_.Value -eq 'Production - Static')})}
#>

foreach ($RG in $StaticRG){
    Write-Host $RG.ResourceGroupName
    $AllResources = Get-AzResource -ResourceGroupName $RG.ResourceGroupName

    $Name = ''
    $Moveable = ''
    $Owner = ''#$Rg.Tags['Owner']
    $Comments = ''
    foreach ($Resource in $AllResources){
        $ResourceType = $Resource.ResourceType
        if($NonMovableResourceTypesWithinSameTenant -contains $Resource.ResourceType){
            $Moveable = 'False'
            $Comments = "Move operation not supported on this Resource Group because --  $ResourceType -- does not support move"
        }
        else{
            $Moveable = 'True'
            $Comments = "Move operation supported for: $ResourceType"
        }
        $ResultObject += [Resource]::new($RG.ResourceGroupName, $Resource.Name, $Moveable, $Owner, $Comments)
    }
}



$ResultObject | ft -AutoSize -Wrap

$ResultObject | Export-Csv -Path .\MoveableResources.csv -NoTypeInformation

$ResultObject | Where-Object { $_.Moveable -eq 'FALSE'} | ft -AutoSize -Wrap

$ResultObject | Group-Object -Property ResourceGroupName

$StaticRG | Group-Object -Property ResourceGroupName

——————– 

############################################################################################################
#Snippet to create RG
############################################################################################################

Set-AzContext -SubscriptionId 'nnnnnnnn-nnnn-eeee-eeee-wwwwwwwwwwww' #new-subscription
foreach ($RG in $StaticRG){
    Write-Host $RG.ResourceGroupName
    New-AzResourceGroup -Name $RG.ResourceGroupName -Location $RG.Location -Tag $RG.Tags -WhatIf:$false -Confirm;$false
}

——————- 

############################################################################################################
#Snippet to move resources
############################################################################################################

$RGsToBeMoved = @('MyRG')

$NewSubscriptionId = 'nnnnnnnn-nnnn-eeee-eeee-wwwwwwwwwwww' #new-subscription
Set-AzContext -SubscriptionId 'oooooooo-oooo-llll-llll-dddddddddddd' #old-subscription

foreach($RG in $StaticRG){
    if($RGsToBeMoved -contains $RG.ResourceGroupName){
    $ResourceIdArray = Get-AzResource -ResourceGroupName $RG.ResourceGroupName | Where-Object {!($_.ParentResource)} | Foreach {"$($_.ResourceId)"} 
    Move-AzResource -DestinationResourceGroupName $RG.ResourceGroupName -DestinationSubscriptionId $NewSubscriptionId -ResourceId $ResourceIdArray `
        -WhatIf:$true -Verbose -Debug
    }
}

—————————– 

############################################################################################################
#Set Delete Lock on RG
############################################################################################################

Set-AzContext -SubscriptionId 'nnnnnnnn-nnnn-eeee-eeee-wwwwwwwwwwww' #new-subscription


foreach ($resourcegroup in $StaticRG){
    $resourcegroupname = $resourcegroup.ResourceGroupName
    Write-Host "RG Name $resourcegroupname"
    if(Get-AzResourceLock -ResourceGroupName $resourcegroupname -AtScope){
        Write-Host "Lock Present Already"
        Set-AzResourceLock -LockName "Do-Not-Delete" -LockLevel CanNotDelete -ResourceGroupName $resourcegroupname -LockNotes 'contact it@renounced'
        Get-AzResourceLock -ResourceGroupName $resourcegroupname -AtScope | ft -AutoSize -Wrap
    }
    else{
        Write-Host "Setting the lock"
        Set-AzResourceLock -LockName "Do-Not-Delete" -LockLevel CanNotDelete -ResourceGroupName $resourcegroupname -LockNotes 'contact it@renounced'
        #Set-AzResourceLock -LockName "Read-Only" -LockLevel ReadOnly -ResourceGroupName $resourcegroupname
    }
}

———————— 

############################################################################################################
# Delete RGs
############################################################################################################

Set-AzContext -SubscriptionId 'oooooooo-oooo-llll-llll-dddddddddddd' #old-subscription

$StaticRG | Foreach-Object { 
    $resourceGroupName = $_.ResourceGroupName
    $RG = Get-AzResourceGroup -Name $resourceGroupName -ErrorVariable notPresent -ErrorAction SilentlyContinue
    if($notPresent){
        Write-Host $resourceGroupName ' NOT PRESENT'
    }
    else{
        $resources = Get-AzResource -ResourceGroupName $resourceGroupName
        $resourceCount = ($resources).Count
        Write-Host $resourceGroupName ' Resource Count: ' $resourceCount
        if ($resourceCount -eq 0){
                    Write-Host $resourceGroupName ' CAN BE DELETED'
                    $locks = Get-AzResourceLock -ResourceGroupName $resourceGroupName
                    if($locks){
                        Write-Host 'Locks found! REMOVING LOCKS'
                        $locks | Foreach-Object { Remove-AzResourceLock -LockName $_.Name -ResourceGroupName $resourceGroupName -Force}
                    }
                    Remove-AzResourceGroup -Name $resourceGroupName -Force
                    Write-Host $resourceGroupName ' DELETE ISSUED'
                    } 
                }
            }

Written by gmaran23

February 5, 2021 at 4:16 pm

Posted in Azure, powershell

Tagged with , , , , , ,

Installing, Managing, and Running SONARQube Community on EC2

leave a comment »

Access a pdf version of this content with rich formatting here – https://renouncedthoughts.files.wordpress.com/2021/02/installingmanagingandrunningsonarqubecommunityonec2-v1.3.pdf

Credits: Mohammed Siyam, Dave Kuppers

Contents

Installing the SonarQube Server. 2

Installing Prerequisites. 2

Installing Java. 2

Configuring Java. 3

Installing Postgresql3

Configuring Default Postgres Identity. 4

Configuring Postgres User for SonarQube. 6

Configuring Authentication Methods for Postgres. 7

Configuring Data Directory for Postgresql9

Configuring Postgres Database and a Postgres User for SonarQube. 11

Downloading and Installing SonarQube server. 11

Creating SonarQube server identity. 12

Configuring SonarQube server. 12

Change System level memory and file system limits for SonarQube Server. 13

Creating systemd files to run ServerQube as a background service. 14

Running a SonarQube Scan with Sonar Runner. 15

Git cloning required respositories. 15

Installing and Configuring node.js to run analysis via sonar-scanner. 15

Installing and Configuring sonar-scanner. 16

Running source code analysis. 16

Updating SonarQube Server. 17

Administering SonarQube Server. 17

Creating and Managing users. 17

Enforcing Authentication. 18

Troubleshooting. 18

 

 

 

 

 

 

 

Installing the SonarQube Server

 

Installing Prerequisites

 

SonarQube has the following prerequisites

1.      Java – as the runtime

2.      Postgres – as the data store

3.      Elastic Search – as the index engine

4.      Node.js – required by sonar-scanner to scan javascript code

5.      Sonar-scanner – the cli used to scan the source code in a directory

Detailed: https://docs.sonarqube.org/latest/requirements/requirements/

 

Installing Java

 

Verify the current JDK version

java -version

Prerequisites mention java version 11, so if java runtime (jre) is version 11, then the prerequisite is met.

However, the current version of java according to the date of this writing is version 14, so install version 14 of openjdk.

Download the latest openjdk from https://openjdk.java.net/install/

wget https://download.java.net/java/GA/jdk14.0.1/664493ef4a6946b186ff29eb326336a2/7/GPL/openjdk-14.0.1_linux-x64_bin.tar.gz

tar -xvf openjdk-14.0.1_linux-x64_bin.tar.gz

cd in to jdk-14.0.1

ls to see if bin directory exists

cd in to bin

ls to see if java exists

./java -version

 

should display the current version, that is 14.

Recursively copy the jdk to /usr/lib/jvm

sudo cp -r jdk-14.0.1 "/usr/lib/jvm"

 

Verify copy

ls "/usr/lib/jvm/jdk-14.0.1/" -l

Remote the temporary unzip and download directories

rm -rf jdk-14.0.1

rm openjdk-14.0.1_linux-x64_bin.tar.gz

 

Configuring Java

 

Adjust the alternatives to set the current version of java (openjdk 14)

alternatives –list

sudo alternatives –install "/usr/bin/java" java "/usr/lib/jvm/jdk-14.0.1/bin/java" 3

sudo alternatives –config java

Choose the java 14 from alternatives

Later, verify

java -version

openjdk version "14.0.1" 2020-04-14

OpenJDK Runtime Environment (build 14.0.1+7)

OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

 

Also set alternatives for jar and javac to the latest openjdk 14, just in case

sudo alternatives –install /usr/bin/jar jar /usr/lib/jvm/jdk-14.0.1/bin/jar 1

sudo alternatives –set jar /usr/lib/jvm/jdk-14.0.1/bin/jar

sudo alternatives –install /usr/bin/javac javac /usr/lib/jvm/jdk-14.0.1/bin/javac 1

sudo alternatives –set javac /usr/lib/jvm/jdk-14.0.1/bin/

 

Installing Postgresql

 

The current version of postgres is 12 according to the date of this writing.

The postgres builds from postgres itself is not supported on amazon linux – https://www.postgresql.org/message-id/flat/15768-35c3af8405f5e346%40postgresql.org

Hence, fall back to version 11, as supplied in the amazon linux extras bundle – https://aws.amazon.com/amazon-linux-2/faqs/#Amazon_Linux_Extras

amazon-linux-extras | grep "post"

NOTE: The livepatch extra is in public preview, not meant for production use

 

  5 postgresql9.6            available    \

  6 postgresql10             available    [ =10  =stable ]

 41  postgresql11             available    [ =11 =stable ]             

 

sudo yum clean metadata

sudo yum install postgresql

 

sudo yum list postgresql*

 

sudo yum install postgresql-server

sudo yum install postgresql-contrib

 

Configuring Default Postgres Identity

 

The default identify that ships with the package to run postgresql is postgres. Change it to postgresuser

sudo adduser postgresuser

sudo passwd postgresuser

 

/database/ is the mount for database files, so the default directory for postgres to be changed

Create a directory /pgsql/data under /database/

sudo mkdir /database/pgsql/data -p

Then change the ownership of /database/pgsql/data to the user postgresuser

sudo chown -R postgresuser:postgresuser /database/pgsql/data

Also change the ownership of the default install directory

sudo chown -R postgresuser:postgresuser /var/lib/pgsql

Delete the default postgres user

sudo userdel -r postgres

Look at the default system unit file for postgresql and extend the unit file configuration to override the default postgres user to postgresuser

cat /lib/systemd/system/postgresql.service

 

 

Extending systemd service unit files are extensively documented at  – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-managing_services_with_systemd-unit_files#sect-Managing_Services_with_systemd-Unit_File_Modify

 

Create a directory postgresql.service.d under /etc/systemd/system

sudo mkdir /etc/systemd/system/postgresql.service.d/

Create a file named overrideuser.conf

sudo touch /etc/systemd/system/postgresql.service.d/overrideuser.conf

 

Edit the overrideuser.conf to provide User and Group information as the newly created postgresuser.

sudo vi /etc/systemd/system/postgresql.service.d/overrideuser.conf

 

[Service]

User=postgresuser

Group=postgresuser

 

:wq!

 

The configuration files from configuration directories in #/etc/systemd/system/ take precedence over the default unit files in #/usr/lib/systemd/system/.

Change the ownership of the /var/run/postgresql where the postgres will create a socket when running

sudo chown -R postgresuser:postgresuser /var/run/postgresql

/var/run/postgresql directory will be erased on every reboot, hence the directory need to be created during reboot with the ownership of the newly created postgresuser.

sudo vi /usr/lib/tmpfiles.d/postgresql.conf

d /var/run/postgresql 0755 postgresuser postgresuser –

:wq!

Failing to correctly modify the ownership and configuration of the /var/run/postgressql might results in errors like below:

sudo systemctl status postgresql.service -l

# FATAL:  could not create lock file

#"/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied

 

 

After the above changes, Enable postgresql.service to be run on system start, then reload the systemd daemon, start the postgresql.service, check the status to see if the service is running.

sudo systemctl enable postgresql.service

 

sudo systemctl daemon-reload

 

sudo systemctl restart postgresql.service

 

sudo systemctl status postgresql.service -l

 

Now postgresql.service should be running under the newly created user identify postgresuser.

 

Configuring Postgres User for SonarQube

 

Login as postgresuser

su – postgresuser

psql –version

should throw a fatal error that “postgresuser” does not exist. This expected behavior us because postgresql creates a template db for every user. So create a create template db for the user postgresuser.

psql

createdb postgresuser

SELECT version();

 

                                                version

———————————————————————————————————-

 PostgreSQL 11.5 on x86_64-koji-linux-gnu, compiled by gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6), 64-bit

(1 row)

 

exit to logout of psql prompt

exit to logout of postgresuser

 

 

 

Configuring Authentication Methods for Postgres

 

Review the authentication methods:

https://www.postgresql.org/docs/11/auth-methods.html

https://www.postgresql.org/docs/11/auth-password.html

 

postgresql by default users ident and trust for local networks and the default authentication methods are found at

sudo cat /var/lib/pgsql/data/pg_hba.conf

 

The password storage mechanism for postgresql users is configured at

sudo cat /var/lib/pgsql/data/postgresql.conf

 

Starting postgresql11, there is support for SHA-256 based passwords with the configuration scram-sha-256. So, change the default password hashing mechanism for postgresql users at postgresql.conf.

 

sudo vi /var/lib/pgsql/data/postgresql.conf

password_encryption = ‘scram-sha-256’

 

:wq!

 

Change the authentication settings in pg_hba.conf

Change peer to trust and ident to scram-sha-256

 

sudo vi /var/lib/pgsql/data/pg_hba.conf

 

 

 

 

 

 

# TYPE DATABASE        USER            ADDRESS                 METHOD

 

# "local" is for Unix domain socket connections only

local   all             all                                     trust

# IPv4 local connections:

host    all             all             127.0.0.1/32            scram-sha-256

# IPv6 local connections:

host    all             all             ::1/128                 scram-sha-256

# Allow replication connections from localhost, by a user with the

# replication privilege.

local  replication     all                                     peer

host   replication     all             127.0.0.1/32            ident

host   replication     all             ::1/128                 ident

 

Reload and start service after authentication change.

 

sudo systemctl status postgresql.service -l

sudo systemctl daemon-reload

sudo systemctl restart postgresql.service

sudo systemctl status postgresql.service -l

 

Verify authentication changes by logging in to psql and checking version.

su – postgresuser

 

psql –version

select * from pg_shadow;

List users, and change the password for the database user named postgresuser.

\du+

\password

After password change the ps_shadow table should show a password for the postgresuser starting with scram-sha-256.

select * from pg_shadow;

Exit from pgsql, and postgresuser.

Change the pg_hba.conf to enforce scram-sha-256 for local connections as well.

 

sudo vi /var/lib/pgsql/data/pg_hba.conf

Change trust to scram-sha-256

 

# TYPE DATABASE        USER            ADDRESS                 METHOD

 

# "local" is for Unix domain socket connections only

local   all             all                                     scram-sha-256

# IPv4 local connections:

host    all             all             127.0.0.1/32            scram-sha-256

# IPv6 local connections:

host    all             all             ::1/128                 scram-sha-256

# Allow replication connections from localhost, by a user with the

# replication privilege.

local  replication     all                                     scram-sha-256

host   replication     all             127.0.0.1/32            scram-sha-256

host   replication     all             ::1/128                 scram-sha-256

 

:wq!

Reload the daemon, and restart the postgresql.service.

sudo systemctl daemon-reload

sudo systemctl restart postgresql.service

sudo systemctl status postgresql.service -l

 

 

As a last change, login to postgres and delete the default user database ‘postgres’.

 

su – postgresuser

psql

 

Enter the newly created password for the database identify postgresuser.

 

List all databases

\l

drop database postgres;

Continue to change the default data directory for postgresql.

 

Configuring Data Directory for Postgresql

 

In the psql prompt, show data_directory displays the current data directory.

show data_directory;

 

 

 data_directory

———————

 /var/lib/pgsql/data

(1 row)

 

Exit psql and postgresuser.

\q

exit

Stop the postgresql.service

sudo systemctl stop postgresql.service

sudo systemctl status postgresql.service -l

 

Move the default data directory to /database mount point preserving the permissions and attributes.

sudo rsync -av /var/lib/pgsql /database          

Change the default data directory configuration in /var/lib/pgsql/data/postgresql.conf

 

sudo vi /var/lib/pgsql/data/postgresql.conf

 

data_directory = ‘/database/pgsql/data’

 

:wq!

After changing the default data directory. Login to psql as postgresuser and check show data_directory;

 

su – postgresuser

psql

 

show data_directory;

    data_directory

———————-

 /database/pgsql/data

(1 row)

 

When the default data directory is moved, override the default systemd unit file configuration to enforce /database/pgsql/data.

sudo touch /etc/systemd/system/postgresql.service.d/overridedatadirectory.conf

sudo vi /etc/systemd/system/postgresql.service.d/overridedatadirectory.conf

 

[Service]

# Location of database directory

Environment=PGDATA=/database/pgsql/data

Environment=PGLOG=/database/pgsql/data/pgstartup.log

 

:wq!

 

Reload daemon, and restart the postgresql.service.

sudo systemctl daemon-reload

sudo systemctl restart postgresql.service

sudo systemctl status postgresql.service -l

 

Configuring Postgres Database and a Postgres User for SonarQube

 

Create a database named sonarqubedb and a user named sonaruser for SonarQube server.

su – postgresuser

psql

create database sonarqubedb;

create user sonarqubeuser with encrypted password ‘ReferToPasswordManager’;

Grant all privileges on database sonarqubedb to sonarqubeuser.

grant all privileges on database sonarqubedb to sonarqubeuser;

 

Downloading and Installing SonarQube server

 

/var/lib/ is the installation directory for sonarqube server. So cd in to /var/lib/

cd /var/lib

Obtain the latest version from https://www.sonarqube.org/downloads/

sudo wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip

Unzip and move contents to the sonarqube directory

sudo unzip sonarqube-8.3.1.34397.zip

sudo mv sonarqube-8.3.1.34397/* sonarqube

 

Remove the default unzip directory and the downloaded .zip file source

sudo rm -r sonarqube-8.3.1.34397

sudo rm -r sonarqube-8.3.1.34397.zip

 

Create sonarqube data directory for elastic search

sudo mkdir /database/sonarqube

sudo mkdir /database/sonarqube/data

sudo mkdir /database/sonarqube/temp

sudo mkdir /database/sonarqube/logs

 

Creating SonarQube server identity

 

Create a user named sonaruser and change ownership of the sonarqube data directory to the sonaruser.

sudo useradd sonaruser

sudo passwd sonaruser

 

sudo chown -R sonaruser:sonaruser /database/sonarqube /var/lib/sonarqube

Configuring SonarQube server

 

Edit sonarqube configuration file present at /var/lib/sonarqube/conf/sonar.properties for postgresql database and webserver settings. Webserver needs to run on port 8080.

sudo vi /var/lib/sonarqube/conf/sonar.properties

And Change the below properties. The properties are spread across the file from the top till the bottom of the file.

sonar.jdbc.username=sonarqubeuser

sonar.jdbc.password=ReferToPasswordManager

sonar.jdbc.url=jdbc:postgresql://localhost/sonarqubedb

 

sonar.web.port=8080

 

sonar.path.logs=/database/sonarqube/logs

 

sonar.path.data=/database/sonarqube/data

sonar.path.temp=/database/sonarqube/temp

 

:wq!

 

Change System level memory and file system limits for SonarQube Server

 

SonarQube server has prescribed minimum memory and file system limits as described in the prerequisites – https://docs.sonarqube.org/latest/requirements/requirements/.

sysctl -w vm.max_map_count=262144

sysctl -w fs.file-max=65536

ulimit -n 65536

ulimit -u 4096

 

Review the current system prerequisites using the commands below and adjust the limits accordingly.

sysctl vm.max_map_count

sysctl fs.file-max

ulimit -n

ulimit -u

 

Adjust the below values only in the minimum configuration is not met by the current operating system.

sudo touch /etc/sysctl.d/99-sonarqube.conf

sudo vi /etc/sysctl.d/99-sonarqube.conf

 

vm.max_map_count=262144

fs.file-max=35536

 

:wq!

 

sudo touch /etc/security/limits.d/99-sonarqube.conf

sudo vi /etc/security/limits.d/99-sonarqube.conf

sonaruser            –       nofile  65536

sonaruser            –       nproc   4096

 

:wq!

 

After the system limit changes, a reboot is required for settings to take effect. So, reboot

 

sudo reboot

 

 

 

 

Creating systemd files to run ServerQube as a background service

 

Create a file named sonarqube.service under /etc/systemd/system with the contents as described below:

 

sudo vi /etc/systemd/system/sonarqube.service

 

[Unit]

Description=SonarQube service

After=syslog.target network.target

 

[Service]

Type=simple

User=sonaruser

Group=sonaruser

PermissionsStartOnly=true

ExecStart=/bin/nohup java -Xms32m -Xmx32m -Djava.net.preferIPv4Stack=true -jar /var/lib/sonarqube/lib/sonar-application-8.3.1.34397.jar

StandardOutput=syslog

LimitNOFILE=65536

LimitNPROC=4096

TimeoutStartSec=5

Restart=always

SuccessExitStatus=143

 

[Install]

WantedBy=multi-user.target

 

Please note the ExecStart parameter takes the .jar file with the version number as argument. During upgrades, this systemd unit file needs it’s ExecStart parameter to be updated accordingly with the more recent version of sonarqube.

 

The above service file is the most suitable way to run sonarqube. However for testing purposes, sometimes if sonarqube needed to be run with the /var/lib/sonarqube/bin/linux-x86-64/sonar.sh file, then the below parameter inside the /var/lib/sonarqube/bin/linux-x86-64/sonar.sh file needs to be changed as sonaruser.

 

#RUN_AS_USER=sonaruser

 

 

Enable the service file on start, start the service, and check status to see if sonarqube is up and running.

 

sudo systemctl enable sonarqube.service

sudo systemctl start sonarqube.service

sudo systemctl status sonarqube.service -l

 

Verify if sonar is running on localhost:8080 by wget.

 

sudo wget http://localhost:8080

 

Running a SonarQube Scan with Sonar Runner

 

Review the Sonar Runner options at https://docs.sonarqube.org/latest/analysis/overview/

Use sonar-scanner as referenced at https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/

 

Git cloning required respositories

 

sudo mkdir -p /database/bitbucket/fantasytravel/

cd /database/bitbucket/fantasytravel/

sudo git clone all or the desired repositories repositories

 

Installing and Configuring node.js to run analysis via sonar-scanner

 

Since node.js is not a part of the Amazon Extras, review the installation instructions at https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-up-node-on-ec2-instance.html

Get the latest version of node.js from

https://github.com/nvm-sh/nvm/blob/master/README.md

 

sudo wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash

. ~/.nvm/nvm.sh

nvm install node

 

Verify node.js installation by checking the version.

node -e "console.log(‘Running Node.js ‘ + process.version)"

Enable node.js for sudoers by creating a symbolic link at /usr/bin

whereis node

node: /home/do.maran/.nvm/versions/node/v14.2.0/bin/node

sudo ln -s /home/do.maran/.nvm/versions/node/v14.2.0/bin/node /usr/bin/node

Reboot to apply changes

sudo reboot

 

 

Installing and Configuring sonar-scanner

 

Get the latest version of sonar-scanner from https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/

cd /database

sudo wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.3.0.2102-linux.zip

sudo unzip sonar-scanner-cli-4.3.0.2102-linux.zip

sudo mv sonar-scanner-4.3.0.2102-linux sonar-scanner

 

Point sonar-scanner to the running url of sonarqube server. The sonar.host.url needs to be reachable from the sonar-scanner location. So, sonar-scanner can not only be run from the same EC2 instance, but also from anywhere in side the VLAN, VPN where the sonarqube server – https://sonarqube.fantasynetwork.com/  is accessible.

sudo vi /database/sonar-scanner/conf/sonar-scanner.properties

sonar.host.url=http://localhost:8080

 

:wq!

 

Make sure sonar-scanner is working by invoking its help.

/database/sonar-scanner/bin/sonar-scanner -h

Remove the downloaded .zip file

sudo rm -fR /database/sonar-scanner-cli-4.3.0.2102-linux.zip

 

Running source code analysis

 

From the root directory of a source code, sonar-runner needs to be invoked to run a source code analysis.

sonar-runner requires basic properties such as project key – which is unique in a sonarqube server.

The basic scan properties can be a part of the sonar-project.properties file in the root directory of a source code, or the basic scan properties can also be passed on to sonar-scanner via the command line.

 

 

 

The below command is an example of running sonar-runner for a source code located at avacado-api-syncer.

cd /database/bitbucket/fantasytravel/avacado-api-syncer

/database/sonar-scanner/bin/sonar-scanner -Dsonar.projectKey=avacado-api-syncer-8D2is5jQ -Dsonar.projectName=avacado-api-syncer -Dsonar.login=sonar-runner -Dsonar.password=ReferToPasswordManager >> /database/run-sonar-scanner-for-all-projects.log

 

sudo if permissions are denied to create the log file named /database/run-sonar-scanner-for-all-projects.log.

sudo bash /database/run-sonar-scanner-for-all-projects.sh

Review the log files at:

cat /database/run-sonar-scanner-for-all-projects.log

Once the analysis is complete, review the results at

https://sonarqube.fantasynetwork.com/

 

Updating SonarQube Server

 

Review the recent version at https://www.sonarqube.org/downloads/

Follow the aforementioned steps, but from an update perspective.

After the update, ensure the sonaruser and the file permissions on the required directories are set appropriately as outlined.

Administering SonarQube Server

 

Creating and Managing users

 

By default, the administrator credentials are admin/admin. This has been changed, refer to the password manager for updated credentials for the default admin user.

sonar-runner is a user created for the sonar-scanner cli, which also is required to be an administrator.

Refer to the password manager for the credentials.

https://sonarqube.fantasynetwork.com/  

Administration è Security èUsers

 

Enforcing Authentication

 

By default SonarQube web ui has authentication turned off, which means anyone hitting the url https://sonarqube.fantasynetwork.com/ would be able to view the scan results and many other information.

Disable anonymous authentication by enforcing authentication.

https://sonarqube.fantasynetwork.com/

Administration è Configuration è General settings è Security è Force user authentication.

 

Troubleshooting

 

Review systemd start up logs by the status command.

sudo systemctl status postgresql.service -l

sudo systemctl status sonarqube.service -l

 

Review postgresql logs at /database/pgsql/data/log

Review sonarqube logs at /database/sonarqube/logs

SonarQube has an elasticsearch subsystem for indexing, the logs are stored with the name es.log

SonarQube has an inbuild java based web server, the logs for the web server are stored at web.log

SonarQube has a compute engine, the logs are stored at ce.log

SonarQube access logs are stored at access.log

 

Success!

Written by gmaran23

February 5, 2021 at 2:24 pm

Posted in AWS, code quality, continuous security, DevOps, devsecops

Tagged with , , , , , , , , , ,