Image Processing - Making Custom Filters - React.js - Part 2

Thanks for reading my first part. Now get ready to dive into another implementation.

Note: Part 1 talks about the fundamental of image processing and then a create a small app using Cloudniary API’s for filters. Checkout the live app here - https://cryptic-sierra-27182.herokuapp.com/, Read here to know more.

The biggest fun in programming is to do the same things in many ways. But everything comes with a cost. As we have used Cloudniary for image processing in the first part of the series, which is a paid solution although a great library to begin with.

Today we will implement some of the things that matter a lot while playing with images in large projects

  1. Smoothing Filters (Some filters and algo’s for it)

  2. Thresholding Filters

  3. Finding contours in the image

Finally, we will use a live camera to capture some images from the webcam live stream.

Let’s start now πŸš€

Libraries

  1. OpenCV: https://www.npmjs.com/package/opencv4nodejs

    a. This one is going to take time if you are a beginner. Setting up a third party system is something to learn.

  1. Node js: https://www.npmjs.com/package/opencv4nodejs

  2. WebCam Connector: https://www.npmjs.com/package/react-webcam

    a. Use this straight implementation of webcam wrapper

  1. API Calling Library

    a. We are using isomorphic-fetch from NPM to call API from react

  1. Frontend: React.js and again material UI see the previous article for more help

    a. https://material-ui.com/

  1. API: Node.js Express.js code written in es6 for API and server-side processing.

Here is the live app link of UI (may take time to load for first time ) - https://peaceful-reef-69295.herokuapp.com/ and Github code link - https://github.com/overflowjs-com/image_app_image_processing_opencvviawebcam_part_2, Please feel free to check it out.

Let’s begin with the UI first β€”

Our aim is to capture the image from the webcam and then apply the filter to them via our own API calls.

  1. Create a new react app:

create-react-app image_app_opencvwebcam

2. Go inside the project and install the dependencies:

cd image_app_opencvwebcam
npm install @material-ui/core β€” save
npm install react-webcam β€” save
npm install β€” save isomorphic-fetch es6-promise

3. Let’s create a containers, components, utlisfolder inside src a folder. The container will contain ImageOps.jsx which is our main entry point and the components will contain Imagefilter.jsx and WebCamCapture.jsx which can be our reusable components. Utils will have our Api.jsAPI wrapper to hit Node.js server.

the directory will look like below

image_app_opencvwebcam
β”œβ”€β”€ README.md
β”œβ”€β”€ node_modules
β”œβ”€β”€ package.json
β”œβ”€β”€ .gitignore
β”œβ”€β”€ public
β”‚   β”œβ”€β”€ favicon.ico
β”‚   β”œβ”€β”€ index.html
β”‚   └── manifest.json
└── src
    β”œβ”€β”€ containers
        β”œβ”€β”€ ImageOps.jsx
    β”œβ”€β”€ components
        β”œβ”€β”€ Imagefilter.jsx
        β”œβ”€β”€ WebCamCapture.jsx
    β”œβ”€β”€ utils
        β”œβ”€β”€ Api.js
    β”œβ”€β”€ App.css
    β”œβ”€β”€ App.js
    β”œβ”€β”€ App.test.js
    β”œβ”€β”€ index.css
    β”œβ”€β”€ index.js
    β”œβ”€β”€ logo.svg
    └── serviceWorker.js

If you have read our first article then you will know what our App.js code will be :)

Let’s check the ImageOps.jsx rendering code:-

import React from 'react';
import Container from '@material-ui/core/Container';
import Grid from '@material-ui/core/Grid';
import Card from '@material-ui/core/Card';
import CardHeader from '@material-ui/core/CardHeader';
import CardContent from '@material-ui/core/CardContent';
import Typography from '@material-ui/core/Typography';
import WebCamCapture from './Components/WebCamCapture';
export default class ImageOpsContainer extends React.Component {
    
 constructor(props) {
   super(props);
   this.state ={
     image_data: null
   };
 }
saveCapturedImage(data) {
  this.setState({ image_data: data });
}
render() {
  return (
    <Container maxWidth="md">
      <Grid container spacing={2}>
        <Grid item xs={12}>
          <Card>
            <CardContent>
               <Typography variant="body" color="textPrimary"   component="p">
                  Image processing part-2
               </Typography>
               <Typography variant="h6" color="textPrimary" component="h6">
                  CAMERA PREVIEW
               </Typography>
               <WebCamCapture saveCapturedImage={(data) =>
               this.saveCapturedImage(data)}/>
            </CardContent>
           </Card>
        </Grid>
      </Grid>
    </Container>
  );
}

Note: We have imported Container, Grid, Card, CardContent, Typography from @material-ui module and WebCamCapture from our own component WebCamCapture.jsx

WebCamCapture, as the name suggests, is used for capturing images from the camera, also we have passed saveCapturedImage function as props which get’s called when we click capture button that we will see in WebCamCapture component. saveCapturedImage function just set the container state with image data.

saveCapturedImage(data) {
  this.setState({ image_data: data });
}

Let’s look now on WebCamCapture component to get more understanding of how the component works

import React from 'react';
import Webcam from 'react-webcam';
import Grid from '@material-ui/core/Grid';
import Button from '@material-ui/core/Button';
export default class WebCamCaptureContainer extends React.Component {
    constructor(props) {
        super(props);
        this.state = {
            videoConstants: {
                width: 1200,
                height: 720,
                facingMode: 'user'
            }
        }
    }
    captureImage() {
          this.props.saveCapturedImage(this.refs.webcam.getScreenshot());
    }
    render() {
   return (
     <div>
      <Grid container spacing={1}>
        <Grid item xs={12}>
          <Webcam
            ref="webcam"
            audio={false}
            // height={350}
            screenshotFormat="image/jpeg"
            // width={350}
            videoConstraints={this.state.videoConstants}
            />
          
        </Grid>
        <Grid item xs={12}>
          <Button variant="contained" align="center" color="primary" onClick={() =>  this.captureImage()} >
            Capture
          </Button>
        </Grid>
      </Grid>
     </div>
   );
    }
}

Here, we have added the Webcam component and Button to capture the current image. Both are in Grid of 12 spaces that means only one component in a row.

On button click, we capture the current image from the webcam as a screenshot and pass that to the props function saveCapturedImagethat we have passed from the ImageOps.jsx.

Note: If you are having trouble at understanding this material code go to β€” https://material-ui.com/components/

Let’s run the project and see what we have built so far vianpm start

React.js UI for Image Processing via Webcam

Let’s move on to filter and code their UI as well.

Smoothing Filters:

Smoothing image or we can say blurring image is very useful in many image operations. The biggest one is reducing noise in the image which is useful if we create a mask or do object detection/face detection or do the processing of any kind of images.

Note: For more depth go to β€” https://docs.opencv.org/2.4/doc/tutorials/imgproc/gausian_median_blur_bilateral_filter/gausian_median_blur_bilateral_filter.html

Now inside components, we will create another component ImageFilter.jsx

import React from 'react';
import Grid from '@material-ui/core/Grid';
import Button from '@material-ui/core/Button';
import Typography from '@material-ui/core/Typography';
import Card from '@material-ui/core/Card';
import CardContent from '@material-ui/core/CardContent';
import { Divider, CardHeader } from '@material-ui/core';
import {api} from '../Utils/Api';
export default class ImageFilters extends React.Component {
 constructor(props) {
        super(props);
        
        this.state = {
            smoothing_effects: [
                {label: "Blur", key: "blur"},
                {label: "Gaussian Blur", key: "gaussian_blur"},
                {label: "Median Blur", key: "median_blur"},
                {label: "Bilateral Filter", key: "median_filter"},
            ],
            render: {}
        }
    }
  applyEffect(effect) {
    api("apply_filter", {
      type: effect,
      data: this.props.image_data
    }).then((data) => {
       const render = this.state.render;
       render[effect] = data;
       this.setState({render});
     });
   }
   getFilterData(effect) {
     if(this.state.render[effect]) {
       return this.state.render[effect];
     }
     return this.props.image_data;
   }
  render() {
    if (!this.props.image_data) {
      return (
        <div/>;
      )
    }
    return (
            <Grid container>
                {this.state[this.props.type].map((effect, i) => {
                    return (
                        <Grid item md={4} key={i}>
                            <Card>
                              <CardHeader title={`${effect.label}   Image`}>
                                </CardHeader>
                                <CardContent>
                                    <img src={this.getFilterData(effect.key)} alt="" height="300px" />
                                    <Button variant="contained" align="center" color="secondary" onClick={() => this.applyEffect(effect.key)} >
                                        Generate
                                    </Button>
                                </CardContent>
                            </Card>
                            <Divider />
                        </Grid>
                     )
                })}
            </Grid>
        )
    }
}

Let’s understand what’s going on with the code.

This component is pretty simple. We have initiated a Grid container and then iterated over state field effects based on props type which can be (smoothing_effects/threshold_effects/contour_effects) to generate each effect card.

Card container is a grid of 4 columns for medium devices and on smaller devices take up whole 12 column space. Now for larger devices, 4 columns will generate 3 columns per row.

On render, we are checking whether the captured image is available or not, if yes, then show the filters to apply.

<Card>
    <CardHeader title={`${effect.label} Image`}>
    </CardHeader>
    <CardContent>
        <img src={this.getFilterData(effect.key)} alt="" height="300px" />
        <Button variant="contained" align="center" color="secondary" onClick={() => this.applyEffect(effect.key)} >
            Generate
        </Button>
    </CardContent>
</Card>

Now we have an image as content and a button to apply the filter on it. The image content is coming from the below function.

getFilterData(effect) {
    if(this.state.render[effect]) {
        return this.state.render[effect];
    }
  return this.props.data;
}

As you can see we are checking the whether effect is saved in the state object. If yes, then render it otherwise just return the original image. Also, on button click, we are calling an api object from Api.js file. In the api object, we are passing the endpoint name and some required parameters. On Successful execution, we will set the state render object and show image with corresponding filters.

applyEffect(effect) {
    api("apply_filter", {
        type: effect,
        data: this.props.image_data
    }).then((data) => {
        const render = this.state.render;
        render[effect] = data;
        this.setState({render});
    });
}

let’s look into the Api.js file in utils

import fetch from  'isomorphic-fetch';
const BASE_API_URL = "http://localhost:4000/"
return fetch(BASE_API_URL+api_end_point,
    {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body:JSON.stringify(data)
    }).then((response) => {
        return response.json();
    });
}

Here we are calling our custom API using fetch API from isomorphic-fetch module.

Note: To read more about the module, check this out https://github.com/matthew-andrews/isomorphic-fetch

In ImageOps.jsx we have created two grids

{this.state.image_data && 
 <Grid item md={12}>
        <CardHeader title={`Captured Image`}>
    </CardHeader>
    <img src={this.state.image_data} alt="" height="300px"/>
</Grid>}
<Grid item xs={12}>
    <Card>
        <CardContent>
            <Typography variant="h6" color="textPrimary" component="h6">
                IMAGE SMOOTH FILTERS
            </Typography>
            <ImageFilters image_data={this.state.image_data} type="smoothing_effects" />
        </CardContent>
    </Card>
</Grid>

The first Grid is used to display the captured image which corresponds to image_data in our state. The second grid is 12 columns grid and contains ImageFilters component and In ImageFilters we are passing two props data image_data and type of effect.

Smoothing Filter UI in React.js

We are now done with the smoothing filter, let’s code the UI for the other two.

Thresholding filters:

Thresholding filter is used in image segmentation which can be used in information extraction. Many processes use this to produce and analyse binary filters. It gives each pixel either white or black. And deciding factor for which pixel will be white or black depends on the algorithm. We will use some of them here.

For information extraction like reading the number from the image, identifying objects or reading text from image these all things use thresholding as one process.

The UI code will mostly be the same with some state update.

In our ImageFilter.jsxcomponent we will add threshold effect to the state.

this.state = {
    smoothing_effects: [
        {label: "Blur", key: "blur"},
        {label: "Gaussian Blur", key: "gaussian_blur"},
        {label: "Median Blur", key: "median_blur"},
        {label: "Bilateral Filter", key: "median_filter"},
    ],
    threshold_effects: [
        {label: "Simple Threshold", key: "simple_threshold"},
        {label: "Adaptive Threshold", key: "adaptive_threshold"},
        {label: "Otsu's Threshold", key: "otasu_threshold"},
    ],
    render: {}
}

and in ImageOps.jsx we will add a new grid for threshold filters.

<Grid item xs={12}>
    <Card>
        <CardContent>
            <Typography variant="h6" color="textPrimary" component="h6">
                THRESHOLDING FILTERS
            </Typography>
            <ImageFilters image_data={this.state.image_data} type="threshold_effects" />
        </CardContent>
    </Card>
</Grid>

Thresholding Filters UI in React.js

Finding Contours:

Contour is creating boundaries over shapes in the image that follow the same color or intensity. We will create a binary threshold image generator and try to map contours (Boundaries around objects) and then draw these boundaries over the image. It can be used in a variety of processes count. Ting object is one of them and plenty more. Its help in filtering the desired object from a set of multiple objects. So let’s check some of its power.

On UI level its same how we did for threshold filter i.e first add the state for contour in ImageFilter.jsx

this.state = {
    smoothing_effects: [
        {label: "Blur", key: "blur"},
        {label: "Gaussian Blur", key: "gaussian_blur"},
        {label: "Median Blur", key: "median_blur"},
        {label: "Bilateral Filter", key: "median_filter"},
    ],
    threshold_effects: [
        {label: "Simple Threshold", key: "simple_threshold"},
        {label: "Adaptive Threshold", key: "adaptive_threshold"},
        {label: "Otsu's Threshold", key: "otasu_threshold"},
    ],
    contour_effects: [
        {label: "Find all contours", key: "find_all_contours"},
        {label: "Find filtered contours", key: "find_filtered_contours"},
    ],
    render: {}
}

and then add the grid in ImageOps.jsx

<Grid item xs={12}>
    <Card>
        <CardContent>
            <Typography variant="h6" color="textPrimary" component="h6">
                CONTOUR FILTERS
            </Typography>
            <ImageFilters image_data={this.state.image_data} type="contour_effects" />
        </CardContent>
    </Card>
</Grid>

and this is how it will look like

Contour Filters UI in React.js

Complete UI as the project will have initial looks as below

Now we have implemented the whole UI for our filters. Now all we need is the working API endpoint to apply filters and see the magic.

Since this article is getting too long, we will implement it in part-3 of Image Processing series. Where we will incorporate OpenCV in Node.js and at last see how everything works end to end.

Get yourself added to our 2500+ people subscriber family to learn and grow more and please hit the share button on this article to share with your co-workers, friends, and others.

For more articles stay tuned to overflowjs.com

Check out articles on Javascript, Angular, Node.js, Vue.js.

Thank you!

Email

About Rakesh Bhatt

Rakesh is a self-learned programmer from around 9 years. He has worked on various programming languages like PHP, java, c#, python, javascript, node js, react, react-native, etc. His forte of working is around the image processing, AR and the building highly scalable web apps.

Subscribe to our email list

More Tags Of Your Interest