Convert AWS DynamoDB Table JSON to Simple PHP Array or JSON

Amazon DynamoDB support these datatype for data

– String
– Binary
– Number
– StringSet
– NumberSet
– BinarySet
– Map
– List
– Boolean
– Null

AWS provides command line tool to SCAN DynamoDB Tables and export as JSON file

# /usr/local/bin/aws dynamodb scan –table-name Users –query ‘Items[*]’ –region us-east-1 –output json

Amazon DynamoDB Formatted JSON using JQ

But AWS DynamoDB exports Table in special JSON format … we can’t import in MySQL without process it to normal JSON file 🙁

AWS DynamoDB JSON

<?php

echo "<pre>";
$tmp = "Users.json";

function aws_dd_json_converter($old) {

    $new = array(); $new1 = array();

	foreach($old as $k=>$v) {
	   foreach($v as $k1=>$v1) {	
       $new[$k1] = isset($v[$k1]['S'])?$v[$k1]['S']:$v[$k1]['N'];  // we can add more datatype here
       }	
	   $new1[] = $new;	
	}
	return $new1;
}

echo "<strong>Original AWS DynamoDB JSON</strong><br>";
echo $line = file_get_contents($tmp);
$array = json_decode($line,true);

$array = aws_dd_json_converter($array);
echo "<br><strong>Simple PHP Array for AWS DynamoDB JSON</strong><br>";
print_r($array);

echo "<br><strong>Simple JSON for AWS DynamoDB JSON</strong><br>";
print(json_encode($array, JSON_PRETTY_PRINT));

?>

Install SSH2 Extension for PHP 7 on CentOS 7

Install gcc, php71w-devel, libssh2 and libssh2-devel on CentOS 7
# yum install gcc php71w-devel libssh2 libssh2-devel

Download php7 pecl-networking-ssh2
# wget https://github.com/Sean-Der/pecl-networking-ssh2/archive/php7.zip

Unzip and Change folder
# unzip php7.zip
# cd pecl-networking-ssh2-php7

Building environment for SSH2 extension
# phpize

# ./configure

Make SSH2 extension make sure build success
# make

Install SSH2 extension
# make install

Write SSH2 extension in PHP 7 extension directory
# vi /etc/php.d/ssh2.ini
extension=ssh2.so

Restart NGINX server and php-fpm
# systemctl restart nginx
# systemctl restart php-fpm

SSH2 PHP7 CentOS7

SSH2 module enabled in PHP 7
# php -m
[PHP Modules]
apc
apcu
bz2
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
intl
json
libxml
mbstring
mcrypt
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
sqlite3
ssh2
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
zip
zlib

[Zend Modules]

Attach And Mount EBS Volume To EC2 Linux Instance

AWS allows to create / attach / mount new EBS volumes to instances for extra storage. Use lsblk command to view available disk devices and other mount points.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 1.6G 6.5G 20% /

– Go to EC2 Console –> Volumes to create new volume of required size and type.

– Select the last created volume and select the “attach volume” to attach to the ec2 instance.

– Now, SSH login to ec2 instance and find list of available disks or mounts.

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 16G 0 disk

Check the volume has data or empty
# file -s /dev/xvdf

Format the volume to ext4 file system type
# mkfs -t ext4 /dev/xvdf

Mount the volume to “/mnt/dir/” directory
# mkdir /mnt/dir/
# mount /dev/xvdf /mnt/dir/ -t ext4

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 1.6G 6.5G 20% /
/dev/xvdf 16G 45M 15G 1% /mnt/dir

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 16G 0 disk /mnt/dir

EBS Automount On Reboot using fstab
# cp /etc/fstab /etc/fstab.bak

Write FSTAB file
# vi /etc/fstab
/dev/xvdf /mnt/dir ext4 defaults,nofail

Check errors
# mount -a

How to unmount volume??
# umount /dev/xvdf

Help: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

Install Ruby Slate NodeJS on CentOS 7

Slate is Beautiful static documentation for your API

Slate

Required

– Linux or OS X or Windows
– Ruby version 2.3.1 or newer
– NodeJS
– Bundler — gem install bundler

# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

Install Ruby 2.4 – CentOS 7

# yum update
# yum groupinstall “Development Tools”
# yum install epel-release

# yum install readline-devel zlib-devel libffi-devel libyaml-devel openssl-devel sqlite-devel

RVM (Ruby Version Manager) Allow you to install and manage multiple ruby environments.

# curl -sSL https://rvm.io/mpapis.asc | gpg --import -
# curl -L get.rvm.io | bash -s stable

# source /etc/profile.d/rvm.sh
# rvm reload

# rvm requirements run

Show available ruby versions
# rvm list known

Install Ruby 2.4

# rvm install 2.4
# rvm use 2.4 --default

Install Ruby Bundler

# gem install bundler

Install NodeJS for JavaScript Run-time Environment

# yum install nodejs npm

# cd /home/centos

Fork / Clone repository

# git clone https://github.com/svnlabs/slate.git

# cd slate

Allow / Open firewall port 4567 for Slate

# systemctl status firewalld

Like firewall-cmd –add-port=[YOUR PORT]/tcp

# firewall-cmd --permanent --add-port=4567/tcp

# systemctl restart firewalld

Run Slate locally

# bundle install
# bundle exec middleman server

Open URL in Browser http://server-ip-address:4567/#introduction

Now edit source files here “/home/centos/slate” as you need 😉

PHP Parse HLS M3U8 TS Links

Input

encoded-03-31-16-thu-jun-2017.m3u8

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=654000,RESOLUTION=568×320,CODECS=”avc1.4d0029,mp4a.40.2″
1080p/encoded-03-31-16-thu-jun-2017.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2849000,RESOLUTION=568×320,CODECS=”avc1.42001f,mp4a.40.2″
720p/encoded-03-31-16-thu-jun-2017.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1033000,RESOLUTION=568×320,CODECS=”avc1.42001e,mp4a.40.2″
360p/encoded-03-31-16-thu-jun-2017.m3u8

encoded-03-31-16-thu-jun-2017.360.m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:YES
#EXT-X-TARGETDURATION:13
#EXT-X-KEY:METHOD=AES-128,URI=”encoded-03-31-16-thu-jun-2017.key”,IV=0x5fd83c919c46159f14ec8da38a4d9ca7
#EXTINF:12.046444,
encoded-03-31-16-thu-jun-201700000.ts
#EXTINF:9.000000,
encoded-03-31-16-thu-jun-201700001.ts
#EXTINF:9.000000,
encoded-03-31-16-thu-jun-201700002.ts
#EXTINF:12.000000,
encoded-03-31-16-thu-jun-201700003.ts
.
.
.
#EXT-X-ENDLIST

Code

<?php

function parseHLS($file, $tag=".ts") {
$return = array();
$i = 0;
$handle = fopen($file, "r");
if($handle) {
while(($line = fgets($handle)) !== FALSE) {
if(strpos($line,"EXT-X-STREAM-INF") !== FALSE) {
if ($c=preg_match_all ("/.*?(BANDWIDTH)(.*?)(,)(RESOLUTION)(.*?)(,)/is", $line, $matches)) {
$return['data'][$i]['bandwidth'] = str_replace("=","",$matches[2][0]);
$return['data'][$i]['resolution'] = str_replace("=","",$matches[5][0]);
}
}

if($tag==".ts") {
if(strpos($line,".ts") !== FALSE) {
$return['data'][$i]['url'] = str_replace(array("\r","\n"),"",$line);
$i++;
}
}

if($tag==".key") {
if(strpos($line,".key") !== FALSE) {  

preg_match( '/URI="([^"]*)"/i', $line, $array );
//print_r( $array[1] ) ;

//$return['data'][$i]['url'] = str_replace(array("\r","\n"),"",$line);
$return['data'][$i]['url'] = str_replace(array("\r","\n"),"",$array[1]);

$i++;
}
}

if($tag==".m3u8") {
if(strpos($line,".m3u8") !== FALSE) {
$return['data'][$i]['url'] = str_replace(array("\r","\n"),"",$line);
$i++;
}
}

}
fclose($handle);
}
return $return;
}


echo "<pre>";

// Get other m3u8 files from m3u8 master manifest 
$u = parseHLS("encoded-03-31-16-thu-jun-2017.m3u8", ".m3u8");

print_r($u);

// Get TS segment files from m3u8 manifest 
$u = parseHLS("encoded-03-31-16-thu-jun-2017.360.m3u8", ".ts");

print_r($u);

// Get key file from m3u8 manifest 
$u = parseHLS("encoded-03-31-16-thu-jun-2017.360.m3u8", ".key");

print_r($u);

echo "</pre>";

?>

Output

Array
(
[data] => Array
(
[0] => Array
(
[bandwidth] => 654000
[resolution] => 568x320
[url] => 1080p/encoded-03-31-16-thu-jun-2017.m3u8
)

[1] => Array
(
[bandwidth] => 2849000
[resolution] => 568x320
[url] => 720p/encoded-03-31-16-thu-jun-2017.m3u8
)

[2] => Array
(
[bandwidth] => 1033000
[resolution] => 568x320
[url] => 360p/encoded-03-31-16-thu-jun-2017.m3u8
)

)

)

Array
(
[data] => Array
(
[0] => Array
(
[url] => encoded-03-31-16-thu-jun-201700000.ts
)

[1] => Array
(
[url] => encoded-03-31-16-thu-jun-201700001.ts
)

[2] => Array
(
[url] => encoded-03-31-16-thu-jun-201700002.ts
)

[3] => Array
(
[url] => encoded-03-31-16-thu-jun-201700003.ts
)
....
.....
)
)

Array
(
[data] => Array
(
[0] => Array
(
[url] => encoded-03-31-16-thu-jun-2017.key
)

)

)

PHP in Amazon Lambda

Clone or Download this GitHub Repo
https://github.com/svnlabs/aws-lambda-php-template

Make new folder lambda in current working directory of EC2 server
# mkdir lambda

Upload 2 files from zip to Amazon EC2 server Instance in folder lambda

lambda
– index.js
– php (binary)

index.js

'use strict';
exports.handler = function(event, context, callback) {
var exec = require('child_process').exec;

  exec('./php -v', function (error, stdout, stderr) {
    if(stdout){
      console.log('stdout: ' + stdout);
    }
    if(stderr){
      console.log('stderr: ' + stderr);
    }
    if (error !== null) {
      console.log('Exec error: ' + error);
    }
    context.succeed(stdout);
    console.log(stdout);
  });
};

Go inside folder lambda

# cd lambda
# zip -r ../lambda.zip *

Copy lambda.zip for AWS S3 bucket

# aws s3 cp ../lambda.zip s3://bucket-name/

How to install AWS CLI?

Create new Lambda Function from AWS Console with NodeJS Runtime*

Name*: phpv
Runtime*: Node.js 4.3
Code entry type: Upload a file from Amazon S3
S3 link URL*: https://s3.amazonaws.com/bucket-name/lambda.zip

Handler*: index.handler
Role*: Lambda Basic Execution

All other info will be default**

Save & Test

“PHP 5.6.5 (cli) (built: Jul 21 2015 18:56:08) \nCopyright (c) 1997-2014 The PHP Group\nZend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies\n”

Free SSL Certbot

Certbot, It was Let’s Encrypt Client previously. Automatically enable HTTPS on your website with EFF’s Certbot, deploying Let’s Encrypt certificates.

Server Support

– Apache
– Nginx
– Haproxy
– Plesk

OS Support

– Web Hosting Service
– Debian 7 (wheezy)
– Debian 8 (jessie)
– Debian 9 (stretch)
– Debian testing/unstable
– Debian (other)
– Ubuntu 17.04 (zesty)
– Ubuntu 16.10 (yakkety)
– Ubuntu 16.04 (xenial)
– Ubuntu 14.04 (trusty)
– Ubuntu (other)
– Gentoo
– Arch Linux
– Fedora 24+
– CentOS 6
– RHEL 6
– CentOS/RHEL 7
– FreeBSD
– OpenBSD 5.9
– OpenBSD 6.0+
– OpenBSD (other)
– macOS
– Devuan Jessie 1.0
– Devuan (other)
– Other UNIX
– Non-UNIX

Install on Ubuntu 17.04 (zesty) Apache

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install python-certbot-apache

Get Started https://certbot.eff.org/#ubuntutzesty-apache

$ sudo certbot --apache

$ sudo certbot --apache certonly

Automating renewal

$ sudo certbot renew --dry-run

$ sudo crontab -e

30 1 * * * /usr/bin/certbot renew --quiet

It will run renewal command at 1:30 am, every day.

PHP Warning: PHP Startup: Unable to load dynamic library /usr/lib64/php/ modules/zip.so

CentOS 7 with PHP

# php --ini
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/zip.so’ – /usr/lib64/php/modules/zip.so: undefined symbol: php_pcre_exec in Unknown on line 0
Configuration File (php.ini) Path: /etc
Loaded Configuration File: /etc/php.ini
Scan for additional .ini files in: /etc/php.d
Additional .ini files parsed: /etc/php.d/10-opcache.ini,
/etc/php.d/20-bcmath.ini,
/etc/php.d/20-bz2.ini,
/etc/php.d/20-calendar.ini,
.
.
.

Install Apache, MySQL, PHP 5.6 on Centos 7

Solution

# yum install php-zip

# service httpd restart
Redirecting to /bin/systemctl restart httpd.service

PHP Check Loaded Modules

# php -m
[PHP Modules]
apc
apcu
bcmath
bz2
calendar
Core
ctype
curl
date
dom
ereg
exif
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
igbinary
json
libxml
mbstring
mcrypt
memcache
memcached
mhash
mongo
msgpack
mysql
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
pgsql
Phar
posix
readline
recode
redis
Reflection
session
shmop
SimpleXML
sockets
SPL
SQLite
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

PHP Check Loaded ini files
# php --ini

# php -v
PHP 5.6.31 (cli) (built: Jul 6 2017 08:06:11)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Install Apache, MySQL, PHP 5.6 on Centos 7

Install Apache

# yum update
# yum install httpd
# systemctl start httpd.service

Now Browser your Apache Server http://Server_IP_address/

Enable Apache to start at server boot

# systemctl enable httpd.service

Install Mariadb MySQL

# yum install mariadb-server mariadb
# systemctl start mariadb
# mysql_secure_installation

Enable MySQL to start at server boot

# systemctl enable mariadb.service

Install PHP 5.6

Install EPEL repository

# rpm -Uvh http://vault.centos.org/7.0.1406/extras/x86_64/Packages/epel-release-7-5.noarch.rpm

Note: Make sure you installed “epel-release-7-5.noarch.rpm” else google it for different location …. search on Google “index of” + epel-release-7-5.noarch.rpm

Install remi repository

# rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-7.rpm

Note: Make sure you installed “remi-release-7.rpm” else google it for different location

Enable remi

# yum --enablerepo=remi,remi-php56 install php php-common

Install php 5.6 on Centos 7

# yum --enablerepo=remi,remi-php56 install php-cli php-pear php-pdo php-mysql php-mysqlnd php-pgsql php-sqlite php-gd php-mbstring php-mcrypt php-xml php-simplexml php-curl php-zip

# systemctl restart httpd.service
# chown -R apache:apache /var/www/html/

# chmod -R 775 /var/www/

<?php phpinfo(); ?>

http://Server_IP_address/phpinfo.php

CentOS 7 + SELinux + Apache + PHP write/access permission

CentOS 7 have SELinux, it is security enhancement to Linux which allows users more control over access control.

SELinux make problem to access or write files or directories

– Can’t serve files on directory
– Can’t write to file

SELinux is blocking the read/write operations

# chcon -Rv --type=httpd_sys_rw_content_t /var/www/html/ (if you want to allow full web root)

# chcon -Rv --type=httpd_sys_rw_content_t /var/www/html/uploads/ (Or just uploads folder)

Ownership
# chown apache:apache -R /var/www/html/

Recursive File permissions
# find . -type f -exec chmod 0644 {} \;

Recursive Directory permissions
# find . -type d -exec chmod 0755 {} \;

Resursive SELinux serve files off Apache
# chcon -t httpd_sys_content_t /var/www/html/ -R

Allow write to specific directories
# chcon -t httpd_sys_rw_content_t /var/www/html/logs/ -R
# chcon -t httpd_sys_rw_content_t /var/www/html/uploads/ -R

httpd_sys_content_t – allow Apache to serve contents
httpd_sys_rw_content_t – allow Apache to write to those path.