<p>Datastructures are essential building blocks upon which any programs are constructed. Each data-structure offers a unique method of arranging data, So It can be retrieaved effectively. <a href="https://docs.python.org/3/library/index.html">Python's Standard Library</a> contains a large collections of data structures.</p>

Guide to Data Structure in pythonverse


Recently I had a talk with my friends about how they want to rewrite their entire company codebase with Rust, given their most of the codebases are bunch of mircro-services and few bit of AI models. Since I have had fair bit of participation on writing code on both Rust and Go. Here is my understanding of these languages and when to use these languages to get most out of it.

## Table of contents

## `Rust` and `Go` : Caught between a rock and hard place are you ?

If I must say something, there is no shortage of languages to choose from, each with its own strengths and weaknesses.I think there are around 20 languages currently, that got fair share of market in current world, each can do lots of great things and great part of uniqueness in them. But given Rust and Go ( also known as Golang ),both are relatively new and popular programming languages that have been gaining lof of traction since their release to public world. I'll try to dive into the differences between Rust and Go and possibly we will figure out when to use these languages.

## What is `Rust` mostly known for ?

- **Memory Safety** : Talking about memory safety, Rust is at top of the chart and Its the best thing that rust offers to its users. Rust's `ownership` system ensures memory safety without the overhead of garbage collection, avoiding the likelihood of common programming errors like null pointer exception and buffer overflows.

- **Concurrency** : Built-in support for concurrency with guaranteed Type Safety, enabling developers to write parallel code that is efficient, safe and easier to maintain without fear.

- **Performance** : It compiles to machine code, which can be optimized for performance, making it ideal for systems programming and performance-critical aplication.

- **C Interoperability** : Rust can easily with C, allowing developers to leverage libraries and gradually migrate legecy codebases.

## What I don't like about `Rust` ?

- **Steep Learning Curve** : Concepts like ownership, lifetimes, borrow and smart pointers takes significant time and effort to master, and can be challenging for newcomers to the field.

- **Compilation Time** : Rust is compiled language. So, It needs to compile Before converting the code into working app and during compile time, Rust compiler try to do all the heavey lifting i.e. borrow checking, make sure app is type saftey, make sure there will be no memory leaks at the runtime and everything. Hence, for fairly complex and big application written in rust , it has ridiculous compilation time.

## What is `Go` mostly known for ?

- **Simplicity** : Go's syntax and design principles principles prioritize simplicity and readability, making it easy for developers to write and maintain code.

- **Concurrency** : Go's built-in support for goroutines and channels simplifies concurrent programming, enabling developers to write scalable applications.

- **Compilation speed** : Go compiles quickly, improving developer productivity and speeding up the development process.

- **Strong Ecosystem** : Go has a growing ecosystem of libraries, tools, and frameworks that make it easy to develop web applications, microservices, and command-line utilities.

## What I don't like about `Go` ?

- **Garbage Collection** : Go's garbage collector can introduce latency and performance overhead in certain scenarios.

- **Error Handling** : Go's error handling way is still in pre-mature phase and it makes code hard to understand.

## Rust vs Go: Which is Right for your application ?

When deciding between Rust and Go, consider the specific requirements of your project.

### **When To use Rust**

- You need a high level of control over system resources, with an emphasis on safety and concurrency
- Systems programming, where performance and safety are critical
- WebAssembly development, enabling high-performance web applications
- Cryptography and security, where memory safety is cucial

### **When to use Go**

- Simplicity, readability, and scalability are your primary concerns
- If you have a choice of converting your application code base to rust or Go ( Use Go instead )
- Microservices, where its concurrency support enables scalable, performant applications.
- Command-line utilites, where its simplicity and fast compilation times are beneficial
- Networking and Distributed sytems, where its concurrency model simplifies complex interactions

## What future holds for them

Both Rust and Go have bright futures in the tech sector, with each language carving out its own niche.

### Rust's Future

Rust's focus on safety and performance makes it an attractive choice for critical infrastructure and systems programming. As more companies prioritize security and reliability, Rust is poised to gain adoption in various sectors, such as aerospace, finance, and healthcare. Additionally, Rust's support for WebAssembly will continue to drive its adoption in web development.

### Go's Future

Go's simplicity and strong support for concurrent programming make it well-suited for modern, scalable applications. As microservices and cloud-native development continue to grow in popularity, Go will likely see increased adoption in these areas. Additionally, Go's growing ecosystem and ongoing language improvements will continue to make it an attractive choice for a wide range of applications.

## Finally

In summary, Rust and Go are both powerful languages with unique strengths and weaknesses. Rust is ideal for systems programming and performance-critical applications, while Go excels in simplicity and scalability for web applications, microservices, and distributed systems. By understanding the differences between these languages and the scenarios in which they excel, developers can make informed decisions about which language is best-suited for their needs.

## REFERENCES

- [Rust Ownership's](https://doc.rust-lang.org/stable/book/ch04-00-understanding-ownership.html)
- [Why use Golang](https://www.uptech.team/blog/why-use-golang-for-your-project)
- [Rust to webassembly](https://surma.dev/things/rust-to-webassembly/)
- [Go for Begineers](https://www.freecodecamp.org/news/go-beginners-handbook/)

## How To Get Started with Rust for Beginners ?

- [Comprehensive Rust By google](https://github.com/google/comprehensive-rust)
- [Official Rust Book](https://doc.rust-lang.org/stable/book/title-page.html)
- [Official Rust Book with examples](https://doc.rust-lang.org/rust-by-example/)
- [Official Rust repo for Rust learners](https://github.com/rust-lang/rustlings/)
- [Awesome Rust learning Resources by Awesome People **Github Repo**](https://github.com/ctjhoa/rust-learning)

## How to Get Started With Golang ?

- [Official Golang documentation](https://go.dev/doc/)
- [Golang Book](https://www.golang-book.com/)
- [50 sates of Go - flaws and common mistakes for new Golang Developers](http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in-go-golang/index.html)
- [Basics of Golang](https://golangbot.com/learn-golang-series/)
- [Another Learning basics of Go webpage](https://miek.nl/go/#packages)
- [Building Web Apps with Go](https://www.openmymind.net/The-Little-Go-Book/?ref=hackr.io?ref=hackr.io)


<p>Rust for developing systems and performance-critical applications, Go for implicity and scalability of mircro-services and distributed systems.</p>

Which would I choose Rust or Go ? `Caught between a rock and a hard place`


React is a popular library for building user interfaces, and one of its most powerful features is the ability to manage component state and side effects. Today, we'll explore the useEffect hook, which allows us to handle side effects in functional components. This guide will cover the basics of useEffect, provide code examples to demonstrate its use in various scenarios, and help you understand when and how to use it.

One of the most crucial ideas for knowing React today is how the 'useEffect' Hook functions. Consider that you have been using React for a while. In that instance, it is especially important to comprehend the differences between using useEffect and using the lifecycle methods of class-based components.

In the React Hooks period, it is crucial to comprehend how to call side effects from within functional components with useEffect. At first, dealing with the useEffect Hook's side effects may seem difficult, but gradually everything will make sense.

## Table of contents

## What are side effects?

Before we dive into how to use useEffect, let's discuss what side effects are in React.In simple terms, `side effects` is any action that occurs outside the scope of the component's `render` method. Such as

1. Fetching data from API
2. Setting up or cleaning event listeners
3. updating the DOM directly
4. Subscribing to a data stream

Those side effects can cause issues in your application if they are not handled properly.For example, if you fetch data
from an API inside a component every time it re-renders, you may end up making too many API calls, which can slow down your application and increase the cost of running it.

## Using `useEffect`

he `useEffect` hook is part of React's [Hooks API](https://react.dev/reference/react) and used to manage side effects in functional component of react.It accepts two arguments:

1. A function that represents the side effects to be performed
2. An optional array of dependencies that determines when the effect should be executed.

```js
useEffect(
  () => {
    // Side effect to be performed
  },
  [
    /* dependencies */
  ]
);
```

### simulating `ComponentDidMount`

Consider following example:

```js
import { useState, useEffect } from "react";

function App() {
  const [data, setData] = useState(null);

  useEffect(() => {
    fetch("https://api.example.com/data")
      .then(response => response.json())
      .then(data => setData(data));
  }, []);

  return (
    <div>
      {data ? (
        <ul>
          {data.map(item => (
            <li key={item.id}>{item.name}</li>
          ))}
        </ul>
      ) : (
        <p>Loading data...</p>
      )}
    </div>
  );
}
```

Above , Inside useEffect hook, we use the fetch function to make a GET request to an API. When the response comes back, we parse the JSON data and update the `data` state variable using the `setData` function.

The dependency array in `useEffect` is the array of values that the effect depends on which means if any of these values change, the effect will re-run or it will keep tracks of object in dependencies array and if any of the object in dependency array don't match with it previous version of data it will re-run the side effects written in `useEffect` hook.

But, In above case the dependency array is empty which means the effect will only run once in its lifetime ( when it is mounting the component ). In terms of class based react, it is exact way of setting states on `ComponentDidMount` lifecycle method.

### Simulating `ComponentDidUpdate`

Consider below example:

```js
import { useState, useEffect } from "react";

function App({ userID }) {
  const [data, setData] = useState(null);

  useEffect(() => {
    fetch(`https://api.example.com/data?user_id=${userID}`)
      .then(response => response.json())
      .then(data => setData(data));
  }, [userID]);

  return (
    <div>
      {data ? (
        <ul>
          {data.map(item => (
            <li key={item.id}>{item.name}</li>
          ))}
        </ul>
      ) : (
        <p>Loading data...</p>
      )}
    </div>
  );
}
```

In above example, dependency array has one item which is also a props to the component. Hence, `useEffect` will track the changes on `userID` props, re-run the side effects present inside the useEffect when value of `userID` changes. Hence, It acts as a `componentDidUpdate` lifecycle method.

### Simulating `componentWillUnmount`

Consider below example:

```js
import { useState, useEffect } from "react";

function App() {
  const [showComponent, setShowComponent] = useState(true);

  useEffect(() => {
    console.log("component mounted!");

    return () => {
      console.log("Component unmounted");
    };
  }, []);

  return (
    <div>
      <button onClick={() => setShowComponent(!showComponent)}>
        {" "}
        Toggle Component
      </button>
      {showComponent && <ChildComponent />}
    </div>
  );
}

function ChildComponent() {
  useEffect(() => {
    console.log("Child component mounted");
    return () => {
      console.log("Child component unmounted");
    };
  }, []);

  return (
    <div>
      <h2>Child Component</h2>
    </div>
  );
}
```

In this example, we define a state variable called 'showComponent' and a function called 'setShowComponent' that we use to toggle the visibility of a child component.

We use useEffect to log a message to the console when the component mounts and a different message when it unmounts. We also define a similar effect inside the child component.

When we toggle the visibility of the child component using the button, we can see the messages in the console. When the child component is unmounted, the cleanup function is called which is similar functionality to the `componentWillUnmount` lifecycle method.

## Conclusion

the useEffect hook is a powerful tool in React that allows you to handle side effects in your components. By using it correctly, you can make your code more efficient and easier to maintain.

Remember, useEffect takes two arguments: a function and an array of dependencies. The function is the side effect you want to perform, and the dependencies are variables that the effect depends on.

You can use useEffect to fetch data from an API, update the title of the page, or set up timers and intervals. You can also use useEffect to return a cleanup function that cleans up any resources used by the effect.

It's important to pass the correct dependencies to useEffect to avoid unnecessary re-renders and memory leaks.

## Tips and Tricks on using useEffect effectively:

- **Use it for handling side effects**: useEffects is designed to handle side effects and you should only use it that way.If you need to modify the state of your component, you should use useState instead.

- **Seperate concerns**: Use multiple useEffect hooks to separate unrelated side effects. This makes your code easier to understand and maintain.

- **Optimize performance**: Be mindful of the dependencies you pass to useEffect. Provide an array of dependencies that accurately represent when the effect should be executed to avoid unnecessary updates.

- **Avoid infinite loops**: When using a state variable as a dependency, make sure your effect does not cause an infinite loop by updating that state variable on every render.

- **Don't forget the cleanup**: Always provide a cleanup function when your side effect requires it to avoid memory leaks and unexpected behavior.


<p>UseEffect hook is a powerful and versatile tool in react that allows you to handle side effects in your components.</p>

Understanding useEffect in React - A Beginner's Guide


Before deep diving into the serverless stacks provided by AWS , we need to know what serverless really means.

## What is serverless ?

Serverless is a cloud-native development model that allows developers to build and run applications without having to manage servers. The servers still gonna be involving while building serverless, but they are abstracted away from app development. A cloud provider like ( AWS, GCP & AZURE ) handles the routine work of provisioning, maintaining , and scaling the server infrastructure. Developers can simply package their code in containers for deployment.

Once deployed, serverless apps respond to demand and automatically scale up and downs as needed. Serverless architectures from cloud providers are usually metered on-demand through an event-driven execution model. As a result, **when a serverless function is sitting idle, it doesn’t cost anything**.

## What are the services provided by aws serverless architecture ?

[![serverless stack aws](/images/blog/serverlessstack.png)](https://aadiimages.imgix.net/images/blog/serverlessstack.png)

It provides many back end tasks like computing, databases, storage, processing and many more, this in result , allows the user to focus on his program and allows him to innovate.

- Compute using lambda

- Storage using Amazon S3

- Database using Amazon Dynamodb

- API proxy using amazon API Gateway

- Application Integration using Amazon SNS

- Orchestration using Step Functions

- Analytics using Amazon Kinesis

- Security & Access Control using its identity and Access Management

## Pros of AWS Serverless Computing

- Servers need no attention for installation and maintenance.

- Payment is as per the throughput, making it value for money.

- You can choose appropriate settings as per your products requirements, thus paying only for the functions you use.

- Serverless have in-built availability and fault tolerance. User need not architect for these capabilities since the services running the application provide them by default.

- Write code and deploy with just few steps, and it will be available to the world within few minutes. Thus no need to put any effort into or for creating and managing servers.

## Cons of AWS Serverless Computing

- [Serverless Architecture](https://aws.amazon.com/lambda/serverless-architectures-learn-more/) excecutes commands and functions on temporarily created container. So if a client performs few tasks on your app, the serverless architecture will create a temporary container and will destroy it as soon as the client is finished performing tasks, this results in delays which are also known as cold start.

- As serverless architecture is based on the temporarily created containers, the usable memory is thus limited hindering the processes that require a lot of processing.

- Another issue with Lambda is that it decides which third-party apps can be used to work on it, thus giving up a lot of control over your application.

- Monitoring and Debugging are quite restrictive to what the vendor provides. It is fundamental with Lambda too. It lacks proper operational tools for monitoring and debugging.

- Lack of local testing options.


<p>Benefits of using serverless stacks.</p>

Serverless stacks and Its benefits


In this blog post, we will explore 10 helpful decorators which I regularly use in my projects to extend my mode with extra functionalites. We'll dive into each decorator, look at the code and play with some hands-on examples. If you are also Python developer, these decorators will help you massively in your python projects.

## Table of contents

## @logger

If you are unfamiliar with decorators, consider reading my previous [blog post](https://aadityachapagain.com/posts/understanding-python-decorator-with-its-usage).

logger is a function that takes a function as input and return a function as output.The output function is usually an extended version of the input. In our case, we want the output function to surround the call of the input function with `start` and `end` statements.

Here's the simple implementation of the `logger` decorator.

```python
from functools import wraps

def logger(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    print(f"----- {function.__name__}: start -----")
    output = func(*args, **kwargs)
    print(f"----- {function.__name__}: end -----")
    return output
  return wrapper


# now you can apply logger to any other functions
@logger
def some_func(text):
  # do something here

```

## @lru_cache

This is a built-in decorator that you can import from `functools`.

It caches the return values of a function , using a least-recently-used( LRU ) algorithm to discard the least-used values when the cache is full.

I typically use this decorator for long-running tasks that don't change the output with the same input like querying a database, requesting a static remote web page, or running some heavy processing.

```python

import random
import time
from functools import lru_cache


@lru_cache(maxsize=None)
def heavy_processing(n):
    sleep_time = n + random.random()
    time.sleep(sleep_time)

# first time
%%time
heavy_processing(0)
# CPU times: user 363 µs, sys: 727 µs, total: 1.09 ms
# Wall time: 694 ms

# second time
%%time
heavy_processing(0)
# CPU times: user 4 µs, sys: 0 ns, total: 4 µs
# Wall time: 8.11 µs

# third time
%%time
heavy_processing(0)
# CPU times: user 5 µs, sys: 1 µs, total: 6 µs
# Wall time: 7.15 µs

```

if you want to implement a cache decorator yourself from scratch, here's how you'd do it:

- You add an empty dictionary as an attribute to the wrapper function to store previously computed values by the input function
- When calling the input function, you first check if its arguments are present in the cache. If it's the case, return the result. Otherwise, compute it and put it in the cache.

```python

from functools import wraps

def cache(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        cache_key = args + tuple(kwargs.items())
        if cache_key in wrapper.cache:
            output = wrapper.cache[cache_key]
        else:
            output = function(*args)
            wrapper.cache[cache_key] = output
        return output
    wrapper.cache = dict()
    return wrapper

@cache
def heavy_processing(n):
    sleep_time = n + random.random()
    time.sleep(sleep_time)


%%time
heavy_processing(1)
# CPU times: user 446 µs, sys: 864 µs, total: 1.31 ms
# Wall time: 1.06 s

%%time
heavy_processing(1)
# CPU times: user 11 µs, sys: 0 ns, total: 11 µs
# Wall time: 13.1 µs

```

## @repeat

This decorator causes a function to be called multiple times in a row.

This can be useful for debugging purposes, stress tests, or automating the repetition of multiple tasks.

Unlike the previous decorators, this one expects an input parameter.

```python

def repeat(number_of_times):
    def decorate(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for _ in range(number_of_times):
                func(*args, **kwargs)
        return wrapper
    return decorate


@repeat(5)
def dummy():
    print("hello")

dummy()
# hello
# hello
# hello
# hello
# hello

```

## @timeit

This decorator measures the execution time of a function and prints the result: this serves as debugging or monitoring.

In the following snippet, the `timeit` decorator measures the time it takes for the `process_data` function to execute and prints out the elapsed time in seconds.

```python
import time
from functools import wraps

def timeit(func):
  @wraps(func):
  def wrapper(*args, **kwargs):
    start = time.perf_counter()
    result = func(*args, **kwargs)
    end = time.perf_counter()
    print(f'{func.__name__} took {end - start:.6f} seconds to complete')
    return result
  return wrapper

@timeit
def process_data():
  time.sleep(1)

process_data()

```

## @retry

This decorator forces a function to retry a number of times when it encounters an exception.

It takes three arguments: the number of retries, the exception to catch and retry on, and the sleep time between retries.

```python

import random
import time
from functools import wraps

def retry(num_retries, exception_to_check, sleep_time=0):
    """
    Decorator that retries the execution of a function if it raises a specific exception.
    """
    def decorate(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for i in range(1, num_retries+1):
                try:
                    return func(*args, **kwargs)
                except exception_to_check as e:
                    print(f"{func.__name__} raised {e.__class__.__name__}. Retrying...")
                    if i < num_retries:
                        time.sleep(sleep_time)
            # Raise the exception if the function was not successful after the specified number of retries
            raise e
        return wrapper
    return decorate

@retry(num_retries=3, exception_to_check=ValueError, sleep_time=1)
def random_value():
    value = random.randint(1, 5)
    if value == 3:
        raise ValueError("Value cannot be 3")
    return value

random_value()
# random_value raised ValueError. Retrying...
# 1

random_value()
# 5

```

## @countcall

This decorator counts the number of times a function has been called.

This number is stored in the wrapper attribute `count`.

```python

from functools import wraps

def countcall(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        wrapper.count += 1
        result = func(*args, **kwargs)
        print(f'{func.__name__} has been called {wrapper.count} times')
        return result
    wrapper.count = 0
    return wrapper

@countcall
def process_data():
    pass

process_data()
process_data has been called 1 times
process_data()
process_data has been called 2 times
process_data()
process_data has been called 3 times
```

## @rate_limited

This is a decorator that limits the rate at which a function can be called, by sleeping an amount of time if the function is called too frequently.

```python

import time
from functools import wraps

def rate_limited(max_per_second):
    min_interval = 1.0 / float(max_per_second)
    def decorate(func):
        last_time_called = 0.0
        @wraps(func)
        def rate_limited_function(*args, **kargs):
            elapsed = time.perf_counter() - last_time_called
            left_to_wait = min_interval - elapsed
            if left_to_wait > 0:
                time.sleep(left_to_wait)
            ret = func(*args, **kargs)
            last_time_called = time.perf_counter()
            return ret
        return rate_limited_function
    return decorate

```

> This function hence introduces a slight time overhead between the calls but ensures that the rate limit is not exceeded.

There's also a third-party package that implements API rate limit: it's called **ratelimit**.

```bash
pip install ratelimit
```

To use this package, simply decorate any function that makes an API call:

```python

from ratelimit import limits

import requests

FIFTEEN_MINUTES = 900

@limits(calls=15, period=FIFTEEN_MINUTES)
def call_api(url):
  response = requests.get(url)

  if response.status_code != 200:
    raise Execption('API response: {}'.format(response.status_code))
  return response

```

if the decorated function is called more times than allowed a `ratelimit.RateLimitException` is raised.

To be able to handle this exception, you can use the `sleep_and_retry` decorator in combination with the `ratelimit` decorator.

```python

@sleep_and_retry
@limits(calls=15, period=FIFTEEN_MINUTES)
def call_api(url):
    response = requests.get(url)

    if response.status_code != 200:
        raise Exception('API response: {}'.format(response.status_code))
    return response
```

This causes the function to sleep the remaining amount of time before being executed again.

## @register

If your pyton script accidentally terminates and you still want to perform some tasks to save your work, perform some tasks to save your work, perform cleanup or print a message, I find that the register decorator is quite handy in this context.

```python

from atexit import register

@register
def terminate():
    perform_some_cleanup()
    print("Goodbye!")

while True:
    print("Hello")

```

## @dataclass

The `@dataclass` decorator in Python is used to decorate classes.

It automatically generates special methods such as `__init__`, `__repr__`,
`__eq__`, `__lt__`, and `__str__` for classes that primarily store data. This can reduce the boilerplate code and make the classes more readable and maintainable.

It also provides nifty methods off-the-shelf to represent objects nicely, convert them into JSON format, make them immutable, etc.

The `@dataclass` decorator was introduced in Python 3.7 and is available in the standard library.

```python

from dataclasses import dataclass,

@dataclass
class Person:
    first_name: str
    last_name: str
    age: int
    job: str

    def __eq__(self, other):
        if isinstance(other, Person):
            return self.age == other.age
        return NotImplemented

    def __lt__(self, other):
        if isinstance(other, Person):
            return self.age < other.age
        return NotImplemented


john = Person(first_name="John",
              last_name="Doe",
              age=30,
              job="doctor",)

anne = Person(first_name="Anne",
              last_name="Smith",
              age=40,
              job="software engineer",)

print(john == anne)
# False

print(anne > john)
# True

asdict(anne)
#{'first_name': 'Anne',
# 'last_name': 'Smith',
# 'age': 40,
# 'job': 'software engineer'}

```

## @singledispatch

This decorator allows a function to have different implementations for different types of arguments.

```python

from functools import singledispatch

@singledispatch
def fun(arg):
    print("Called with a single argument")

@fun.register(int)
def _(arg):
    print("Called with an integer")

@fun.register(list)
def _(arg):
    print("Called with a list")

fun(1)  # Prints "Called with an integer"
fun([1, 2, 3])  # Prints "Called with a list"

```

## Conclusion

Decorators are useful abstractions to extend your code with extra functionalities like caching, automatic retry, rate limiting, logging, or turning your classes into supercharged data containers.

Here's a [list](https://github.com/lord63/awesome-python-decorator) of awesome decorators to get inspired.

Thanks for reading!.


<p>Do more thing with less code without compromising on quality</p>

Useful Python decorator to take your code to MARS and beyond


You might have seen python code that has `@<function_or_class>` written immediately above a function or Class.
For Example, `@staticmethod` right above static method or `@classmethod` right above class method. Those are
actually Python decorators. Decorators allow an existing function to be extended without changing the source code.

In this blog, we will comprehend a decorator structure, examine advanced behaviours including drawbacks and ways to overcome it, nested decorators, stacking decorators and eventually some practical applications, develop our own multi-purpose decorators.

## Table of contents

## Writing Decorator

Decorators are often referred to as function wrappers because they accept a function as a paramter and return a modified version of the function that has additional features or capabilites.

```python

def simple_decorator(func):
  def wrapper(*args, **kwargs):
    # do something before function execution
    result = func (*args, **kwargs)
    # do something after function execution
    return result
  return wrapper

```

we can see from the construct above that line 4 is where the function really executes, but we can change what happens before to, during and even after a function execution. Althouh decorators may alter a function's input, output, or behaviour, it is preferable to build them in a way that does not make the function they are wrapping less understandable.

> **Decorators are best used to add common behaviour to multiple functions without modifying every function manually**

## Advanced behaviours of Decorators

### Retain metadata of wrapped function

One major **drawback** of using decorators is that the metadata of function will be destroyed by decorator when actually calling that function in code, because we are returning a wrapper function in place of the original function in the code sample from the previous section, all decorated functions will have their `__name__` and `signature` information changed to wrapper's.

```python

@simple_decorator
def func_add(a, b):
  return a + b

print(func_add.__name__)
# wrapper

```

Technically, this wouldn't impact how the fucntion or decorator was meant to be used, but it's still advisable to avoid any unexpected outcomes when using a decorator. This can be accomplished quickly by decorating the wrapper function with the `@wraps` decorator, as illustrated below.The decorator can still be used in the same way, but now the wrapped function's metadata won't be changed.

```python

from functools import wraps

def sample_decorator(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    # do something before function exec
    result = func(*args, **kwargs)
    # do something after function exec
    return result
  return wrapper

```

### Decorators that accept arguments

Under the hood we all know that decorators are just a function wrapper. So, it is possible for decorator to receive arguments to make make this happen , we just need to slight change in coding our decorator.
We will get decorator that will take arguments, If only we just wrap the existing decorator with another function with arguments we wanna pass to the decorator. Easy right !.

Example below I have a `debug_decorator` that takes parameters and returns a `decorator` decorator that wraps thea original function in a new one. The several nested function in this can make it appear fairly confusing at first but first creating the original decorator before wrapping it to take argumeants will make it easy to code and understand.

```python

from functools import wraps

debug_mode = True


def debug_decorator(debug_mode):
    """Example: Passing arguments to a decorator"""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            if debug_mode:
                print(f"Function called: {func.__name__}")
            result = func(*args, **kwargs)
            return result
        return wrapper
    return decorator


@debug_decorator(debug_mode)
def func_add(a, b):
    return a + b


func_add(1, 2)
# Function called: func_add
# 3
```

### Stacking decorators

Decorators, as previously indicated, permit the extension of current functionalities. To add extra extensions, it is possible to stack several decorators over a function. The decorator will be piled in the same order as tahe order of execution.

One thing to remember is that if time-sensitive decorators are layered, they should be added at last. So that they can accurately reflect the execution of time without being influeced by other decorators. Decorators that measure the execution of time of a function, for instance, should be the last to be executed.

Now, that we are familiar with a decorator's structure and sophisticated behaviours, we can explore their actual use!

## Usage of Decorators

### Measuring execution time of functions

The `timer` decorator can measure the execution time of the wrapped function by recording the start time and end time of the function execution and printing the results to the console.

In the code snippet below, we measure the `start_time` and `end_time` before and after function execution.

```python
import time

from functools import wraps

def timer(func):
    """Example: Measure execution time of function"""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"Execution time: {round(end_time - start_time, 4)}")
        return result
    return wrapper


@timer
def func_add(a, b):
    time.sleep(2)
    return a + b


func_add(1, 2)
# Execution time: 2.0064

```

### Debug with logging

The `logging` decorator can be used to log information to a console or log file and is useful for debugging. Below we'll use the `logging` python package to perform logging.

```python

import logging

from datetime import datetime
from functools import wraps

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def logging(func):
    """Example: Logging with decorator"""

    @wraps(func)
    def wrapper(*args, **kwargs):
        log_time = datetime.today().strftime("%Y-%m-%d %H:%M:%S")
        logger.info(f"{log_time}: {func.__name__} called")
        result = func(*args, **kwargs)
        return result
    return wrapper

```

### Registering Plugins

Decorators don’t have to wrap the function they’re decorating. They can also simply register that a function exists and return it unwrapped. This can be used, for instance, to create a light-weight plug-in architecture:

```python
import random
PLUGINS = dict()

def register(func):
    """Register a function as a plug-in"""
    PLUGINS[func.__name__] = func
    return func

@register
def say_hello(name):
    return f"Hello {name}"

@register
def be_awesome(name):
    return f"Yo {name}, together we are the awesomest!"

def randomly_greet(name):
    greeter, greeter_func = random.choice(list(PLUGINS.items()))
    print(f"Using {greeter!r}")
    return greeter_func(name)

```

The `@register` decorator simply stores a reference to the decorated function in the global `PLUGINS` dict. Note that you do not have to write an inner function or use @functools.wraps in this example because you are returning the original function unmodified.

The randomly_greet() function randomly chooses one of the registered functions to use. Note that the PLUGINS dictionary already contains references to each function object that is registered as a plugin:

```python
>>> PLUGINS
{'say_hello': <function say_hello at 0x7f768eae6730>,
 'be_awesome': <function be_awesome at 0x7f768eae67b8>}

>>> randomly_greet("Alice")
Using 'say_hello'
'Hello Alice'

```

The main benefit of this simple plugin architecture is that you do not need to maintain a list of which plugins exist. That list is created when the plugins register themselves. This makes it trivial to add a new plugin: just define the function and decorate it with `@register`.

If you are familiar with `globals()` in Python, you might see some similarities to how the plugin architecture works. `globals()` gives access to all global variables in the current scope, including your plugins:

```python

>>> globals()
{..., # Lots of variables not shown here.
 'say_hello': <function say_hello at 0x7f768eae6730>,
 'be_awesome': <function be_awesome at 0x7f768eae67b8>,
 'randomly_greet': <function randomly_greet at 0x7f768eae6840>}

```

Using the `@register` decorator, you can create your own curated list of interesting variables, effectively hand-picking some functions from `globals()`.

### Creating a singleton class

> **Singleton class is a design pattern that restricts the instantiation of a class and ensures that only one instance of the class exists.**

Singleton class is helpful when there is a cap on the number of concurrent users who can use a shared resource or when there is only one point of access to a resource. In python singleton class can be created just using decorator.

Single instantiation can be guaranteed by specifically coding singleton classes. But, utilising decorators is a clever method to reuse the code for several classes if there are several singleton classes.

```python
from functools import wraps


def singleton(cls):
    """Example: Create singleton class with decorator"""
    instances = {}

    @wraps(cls)
    def wrapper(*args, **kwargs):
        if cls not in instances:
            instances[cls] = cls(*args, **kwargs)
        return instances[cls]
    return wrapper


@singleton
class SampleClass:
    def __init__(self):
        pass


singleton_class = SampleClass()
singleton_class2 = SampleClass()
print(singleton_class == singleton_class2)
# True

```

Hope, you now have a solid understanding of decorators fundamentals, practical advice and real-world examples. Other applications include the use of decorator for caching, memory management, and timeout precedures. These decorators are more sophisticated, thus using built-in decorators from python or decorators from 3^<sup>rd</sup> party python packages is preferable to creating them from scratch.

Hope In future , I will come up with new blog posts explaining all the really really complicated python decorators.

## REFERENCES

- [https://www.wikiwand.com/en/Singleton_pattern](https://www.wikiwand.com/en/Singleton_pattern)

- [https://peps.python.org/pep-0318/](https://peps.python.org/pep-0318/)

- [https://realpython.com/primer-on-python-decorators/](https://realpython.com/primer-on-python-decorators/)


<p>Upskill your python using awesome decorators</p>

Understanding Python Decorator and Its Usage


Recently, I was working on Big Machine Learning project. The task was to pretraining Large Machine learning models (with parameter in the range of several Billion ). And Normal training approch didn't work ( obviously ).With 8 GPU Volta core machines, it would take several months to complete just 1 epcoh of training, that's the point when i think of distributed training. I was using gcp ( google cloud ) for training models and found out that google already have support for High Performance Computing with Slurm . You can find Minimal working example on slurm from google codelabs here [https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine](https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine).

Through this blog, I will try to explain what is HPC? , Why HPC ?, how can we train large Deep Learning models with slurm.

# What is HPC ?

High Performance Computing (HPC) is the use of supercomputers and parallel processing techniques for solving complex mathematical and computational problems. HPC technology primarily focuses on developing parallel processing algorithms and systems by incorporating both administration and parallel computational techniques. HPC is typically used for solving advanced problems that require a lot time .

# Why HPC ?

When you have loads of data and its processing takes really long time, then the approch **divide et impera** comes at hand.

With HPC, we can divide any job so that every node processes different partitions of the data in parallel, speeding up the execution time.

# Slurm Workload Manager on GCP

To make computing with slurm easier in GCP, Google and SchedMD ( Slurm's Creators ) joined forces and as a result, we can run a Slurm cluster on Google Cloud Platform. We don't have to worry about the parallel computing techniques since slurm takes care of that and GCP takes care of setting up a cluster and providing resources.
Basic architectural diagram of a stand-alone Slurm Cluster in Google Cloud Platform.

[![Slum Architecture in GCP](/images/slurm-archi.png)](https://ibb.co/SPhjRyM)

As we can see in above pictures, Slurm cluster contains three types of nodes: **login**, **controller** and **Compute** node.

- **Login Node** serves as an interface for the user: user should communicate with the cluster exclusively through the login node (starting the job, requiring resources, ...)
- **Controller Node** manages resources and job scheduling for the user.
- **Compute Node** executes the job.

# Setting Up a Slurm Cluster on GCP

Before describing the setup, let us explain in short how does GCP implement a slurm cluster.

In GCP, a cluster is realized as a **deployment**. A deployment is an instantiation of a set of resources that are defined in a configuration. A deployment can contain a number of resources, across a variety of Google Cloud Platform services. When you create a deployment, Deployment Manager creates all of the described resources in the respective Google Cloud Platform APIs.

This brings us to the cluster's nodes. Each node in a cluster is actually a **Virtual Machine**.When a deployment is created, three new virtual machines appear in "VM instances" page, under "Compute Engine". Those VMs are login instances, Controller instances, and compute image instance.

Compute Instance is a bit trickey part. One thing to notice is that deployment does not create compute instance,but exactly one compute image instance even if you request more compute nodes in your cluster. So, if a user requests 10 compute nodes for the cluster, those 10 virtual machines will not be immediately instantiated with the cluster deployment. Here's what is happening. These compute instances are created in the later step when you run a job and request the number of nodes for the job. Then the compute nodes will be allocated, and they will appear in the "VM Instances" page. Shortly after the job is completed, these virtual machines will be deallocated and will disappear from the list. This way a user gets new compute VMs everytime. The fact that deployment create compute image instance rather than compute nodes directly is that, you might not be using compute node all the time and creating compute nodes unnecessarily might affect your billing, so , slurm will create new compute nodes and use compute image instance as a templete to dynamically create new instance during running jobs, so that you will be billed for exact time period your compute node will run.

Below you can see visual representation of the described process:

[![Slum Architecture in GCP](/images/slurm-hpc.png)](https://ibb.co/mFrGDy3)

Finally, let's head to the cluster setup. In this blog post, we will setup Slurm cluster for training Deep Learning Model with several nodes.Customize the information so that they will suit your needs:

1. Launch Google Cloud Shell
2. Check that you already authenticated and that the project is already set to your **PROJECT_ID**:

```bash
$ gcloud auth list

Credentialed accounts:
<your email>

$ gcloud config list project
[core]
project = <PROJECT_ID>

```

1. Clone git repository that contains the Slurm Google Cloud Platform deployment-manager files:

```bash
$ git clone https://github.com/SchedMD/slurm-gcp.git

```

2. Switch to the Slurm deployment configuration directory:

```bash
$ cd slurm-gcp
```

3. Configure the Slurm Deployment YAML file. Provide information that suits your needs. There are plenty more parameters available, they can be found in SchedMD's GitHub repository. Below is the script that was sufficient for my needs.

```yaml

# [START cluster_yaml]
imports:
- path: slurm.jinja

resources:
- name: slurm-cluster
type: slurm.jinja
properties:
    cluster_name            : slurm-job

    zone                    : us-central1-b
    controller_machine_type : n1-standard-2
    controller_disk_type      : pd-standard
    controller_disk_size_gb   : 50
    external_controller_ip    : True

    login_machine_type        : n1-standard-2
    login_disk_type           : pd-standard
    login_disk_size_gb        : 50
    external_login_ips        : True

    compute_image_machine_type  : n1-standard-2
    compute_image_disk_type   : pd-standard
    compute_image_disk_size_gb: 200
    external_compute_ips      : False

    partitions :
    - name           : gpu
        machine_type   : n1-standard-16
        max_node_count : 10
        zone           : us-central1-b
        # cpu_platform           : Intel Skylake
        preemptible_bursting   : True
        compute_disk_type      : pd-standard
        compute_disk_size_gb   : 200
        gpu_type               : nvidia-tesla-v100
        gpu_count              : 1

#  [END cluster_yaml]

```

4. In the Cloud shell Session, execute the following command from the slurm-gcp folder:

```bash
> gcloud deployment-manager deployments create slurm-deployment  --config  slurm-cluster.yaml
```

    This command creates a deployment  named slurm-deployment. The operation can take few minutes to complete.

5. Verify the deployment ( Navigation menu --> Deployment Manager)
6. Verify the cluster's instances ( Navigation menu --> Compute Engine --> VM Instances) There should be login , controller, and compute image instances. Compute image instance will live for small amount of time for setting up compute images, after that it will be off. and Compute Instances show up only when you allocate them for the sbatch job. they disappear shortly the job is completed.
7. Log in to login instance .

   While logging in to the login instances if _Slurm is currently being installed/configured in the background._ Don't install any packages during that time, as that might disrupt the installation process of slurm. Wait for around 10 for slurm to be installed fully and you can log in to login instances and do whatever you want.

# Setting up Deep Learning Environment inside slurm cluster

1. SSH to login instance in your slurm cluster.
2. After sucessfully installing slurm on cluster, ( /home , /apps ) directory of login instances will be shared with all other instances like controller and compute instances. So that all deep learning and python binaries must be installed inside `/home` or `/apps` directory.
3. Installing python inside /home:

```bash
cd ~

sudo yum -y update; sudo yum groupinstall "Development Tools"

sudo yum -y install openssl-devel bzip2-devel libffi-devel

mkdir tmp;cd tmp; wget https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz

tar zxvf Python-3.8.3.tgz; cd Python-3.8.3

./configure --enable-optimizations --prefix=$HOME/opt/python-3.8

make

make altinstall

vi ~/.bashrc
(add line: export PATH=$HOME/opt/python-3.8/bin:$PATH) at the end of .bashrc

source ~/.bashrc
```

```
> which python3.8
(output should be: /home/<username>/opt/python-3.8/bin/python3)

# Now Lets install pytorch

> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

>  python3.8 get-pip.py

> python3.8 -m pip install torch==1.5.0

```

4. Installing CUDA toolkits and Nvidia-drivers

- Download the latest Nvidia CUDA [repository package](https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/) cuda-repo-rhel7-\*.rpm.

```bash
> cd ~
> wget https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.2.89-1.x86_64.rpm
```

- Install the CUDA repository package. This will enable CUDA repository on your CentOS 7 Linux system:

```bash
> sudo rpm -i cuda-repo-*.rpm
```

- Install cuda package from nvidia repository. Pytorch 1.5 works with cuda 10.2, so lets install cuda 10.2. Below command will install cuda and nvidia-drivers inside `/usr/local/cuda-10.2` directory.

```bash
> sudo yum install cuda-10-2
```

- By Now we know that only `/home` and `/apps` directory were shared across all other instances. So lets copy cuda directory from `/usr/local/` to `~/opt/cuda`

```bash
> mkdir ~/opt/cuda
> cp -r /usr/local/cuda-10.2/* ~/opt/cuda/
```

- Export cuda path to Nvidia CUDA binary executables. Open the ~/.bashrc using your preferred text editor and add the following two lines

```bash
export PATH=$HOME/opt/cuda/bin:$PATH
export LD_LIBRARY_PATH=$HOME/opt/cuda/lib64:$LD_LIBRARY_PATH
```

- Now Re-login or execute your updated ~/.bashrc file:

```bash
> source ~/.bashrc
```

- Now Confirm the CUDA installation:

```bash
> nvcc --version
> nvidia-smi #( doesn't work as we dont have gpu in login instances )
```

> By Now your slurm system is ready and can run any training script using `sbatch` scheduling. Below I will discuss on how to compile nvidia-apex library. It takes me 2 days just to make nvidia-apex work on slurm cluster.If you follow my above steps exaclty, I am sure you will be able to install nvidia apex library inside your slurm cluster without any problems.

# Installing Nvidia-Apex library inside your slurm cluster ( Optional )

If you don't know about Apex library, you can visit Nvidia-Apex [github page](https://github.com/NVIDIA/apex) to know more about Apex library. It is a set of utilites to help training model in mixed precision mode. It provide more numerical stable layer norm operations during training model in mixed precision mode, So that you can have both faster and stable training.

For `Nvidia-apex` to sucessfully compile, You need to upgrade your gcc --version to 7.3.0 .

```bash
> sudo yum install centos-release-scl
> sudo yum install devtoolset-7-gcc*
> scl enable devtoolset-7 bash
> which gcc
> gcc --version
(above bash script must output gcc version as 7.3.0)
```

Now We will compile and install Apex-library inside python3.8

```bash

> git clone https://github.com/NVIDIA/apex
> cd apex
> export TORCH_CUDA_ARCH_LIST="6.0;7.0"
> CUDA_HOME=$HOME/opt/cuda python3.8 -m pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

Its takes around 4 - 6 minutes to compile and install apex.

# Distributed Training with pytorch and slurm

I will not give details on how to setup codebase in pytorch for distributed training, Its a huge topic and might need another blog post for it only. Meanwhile, You can learn more about distributed training with pytorch [here](https://pytorch.org/tutorials/intermediate/dist_tuto.html). After you done setting up distributed training codebase in pytorch. You can start training using sbatch scripts.

## Running the training Job Using SBATCH Script

After we have done the cluster setup, preparing deep learning envs and building of the model, the last step is to finally run a job, i.e. to start the training. That is easily done by running sbatch script, which is basically a customized shell script. It effectively has two parts. The first part of the script is specific for the Slurm, it specifies the parameters for the Slurm job scheduler using the SBATCH command. The second part consists of bash (or some other shell) commands that you would normally run in terminal.

Below you will see demo SBATCH script (You need to modify sbatch script according to your needs).

> inside run_training.sh file.

```bash
#!/bin/sh
 #SBATCH --job-name=distributed_training
  #SBATCH --output=slurm_logs/slrm_stdout.%j
  #SBATCH --error=slurm_logs/slrm_stderr.%j
  #SBATCH --partition=gpu
  ## make sure we don't clobber log files if jobs get restarted
  #SBATCH --open-mode=append
  #SBATCH --nodes=2
  #SBATCH --time=24:00:00
  ## make sure we are told about preempts, and jobs running out of time, 60s beforehand
  #SBATCH --signal=USR1@60
  #SBATCH --cpus-per-task=5
  ## srun forks ntasks_per_node times on each node
  #SBATCH --ntasks-per-node=1
  #SBATCH --mem=200G
  #SBATCH --gpus-per-node=1

python3.8 train.py <Your training args here>
```

Execute the sbatch script using the sbatch command line:

```bash
> sbatch run_training.sh
```

Running sbatch will return a Job ID for the scheduled job, for example:

```
Submitted batch job 37
```

To keep track of the job’s state, run squeue and to keep track of the cluster’s state, run sinfo:

```bash
> squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3 gpu hostname <username> CF 0:11 4 slurm-job-compute-0-[0-1]

> sinfo

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
gpu* up infinite 4 mix# slurm-job-compute-0-[0-1]
gpu* up infinite 6 idle~ slurm-job-compute-0-[2-7]
```

# Summary

In this blog post we learned how to setup slurm cluster in GCP ( which is super easy ), setup own deeplearning environment from installing python, pytorch to compiling apex from source and finally run training using sbatch script and keeping track of job state using `sinfo`, `squeue` and `scontrol`.

# REFERENCES

1. [https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine](https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine)
2. [https://cloud.google.com/deployment-manager/docs/deployments](https://cloud.google.com/deployment-manager/docs/deployments)
3. [https://pytorch.org/tutorials/intermediate/dist_tuto.html](https://pytorch.org/tutorials/intermediate/dist_tuto.html)
4. [https://github.com/NVIDIA/apex](https://github.com/NVIDIA/apex)


<p>Faster Training of Large Deep Learning models is much easier than you think with the help of Slurm.</p>

Distributed Training of Deep Learning model with Slurm on GCP


Have you ever worked on large Machine learning projects or research where you have to manage many experimentations ? often during large projects or experiments you have to log every nits and bits of your Machine Learning training. Some of us might already be there.
Recently I was also working on large Machine learning Projects and it was very hard for me to track every experiments logs, visualization, experiments. There were lots of them and It would take lots of time to prepare presentation's of my experiments results to my peers and that's when I learned about wandb.

Wandb is API created by [Weights & Biases](https://www.wandb.com/) to collect, manage and visualize all your Machine learning experiments all at one place. Ohh, wait you must be wandering that **tensorboard** can also do these things right ?

Well, there are lots of things which makes **wandb** a straight winner.

1. Visually aesthetics UI.
2. More manageable and customizable visualization and experiment tracking.
3. Super easy to use API.
4. Analyze system usage metrics alongside runs.
5. Collaborate with team members
6. Run parameter sweeps
7. Keep records of experiments available forever

I think the last point in above list is super cool, which is not feasible to do with tensorboard i.e You have to write huge amount of code by yourself to log these system wide metrics into tensorboard, but with wandb you can do this with just one line of code. and

Lets Dive into wandb API.

# Installation

Installing `wandb` is very easy, just run below command in terminal and you are good to go.

```bash
pip install wandb

pip install wandb --upgrade
```

One thing I noticed during installing wandb is that, if we run only `pip install wandb` it will sometimes only install wrapper class of wandb. So, upgrading wandb library after installing it , worked for me.

# Integration with your python code

You can use `wandb` with any deeplearning framework, either it is pytorch or Tensorflow.But, first you need to have account on [Weights & Biases](https://www.wandb.com/). After you create account on [Weights & Biases](https://www.wandb.com/) , you can get your wandb api keys from [setting](https://app.wandb.ai/settings). After you get your keys, you just need to write couple of code to integrate your deeplearning system with wandb logger.

## Login

First you need to login into your wandb using wandb API KEYS. you can login programetically or from terminal, but login using terminal is advised.

```bash
# This is secret and shouldn't be checked into version control
export WANDB_API_KEY=$YOUR_API_KEY

wandb login
```

To Programatically login you can use following code:

```python
WANDB_API_KEYS = "<your api keys here>"

import wandb

wandb.login(WANDB_API_KEYS) # if you don't have $WANDB_API_KEYS in your env variable (Not Advised)

wandb.login() # if $WANDB_API_KEYS is already been set.
```

## Initializing wandb ( wandb.init )

Below you will see simple wandb integration examples with keras.

```python

# initialize wandb with your project name and optionally with configutations.
wandb.init(project='demo-keras-integration', name = 'first_run'
           config={
              "learning_rate": 0.005,
              "epochs": 25,
              "batch_size": 64,
              "loss_function": "sparse_categorical_crossentropy",
              "architecture": "CNN",
              "dataset": "CIFAR-10",
           }, anonymous='never')
config = wandb.config

# Initialize model like you usually do.
tf.keras.backend.clear_session()
model = Model()
model.summary()

# compile model like you usually do.
# notice use of config.
optimizer = tf.keras.optimizers.Adam(config.learning_rate)
model.compile(optimizer, config.loss_function, metrics=['acc'])

```

`project` is name of your project and your project might have different run, so `name` parameter will distinguish one run from other.
If you run above code and get back to your wandb dashboard it will new project called `demo-keras-integration` and with `first_run` in it.

[![wandb Init](/images/wandb-init.png)](https://ibb.co/hDjwC6s)

Above init configuration will create new run everytime you call `wandb.init` But sometimes that not what you want, If your training is preemptible and might takes days or months to complete then you can resume from previous logged metrics by providing `resume = True` parameter to `wandb.init` and you will also need to set unique id, to distinguish one run from another. Below code will resume your logs with current run in your wandb dashboard even after you run `init` multiple times.

```python

wandb.init(project='demo-keras-integration', name = 'first_run', resume= True,
            id = 'my_first_run'
           config={
              "learning_rate": 0.005,
              "epochs": 25,
              "batch_size": 64,
              "loss_function": "sparse_categorical_crossentropy",
              "architecture": "CNN",
              "dataset": "CIFAR-10",
           }, anonymous='never')
```

## Train with Wandb callback

wandb made easy to log your model metrics into your project space by providing various callback function to log your metrics directly into wandb dashboard, without writing extra code.

```python

from wandb.keras import WandbCallback

# train with our favorite model.fit
# notice WandbCallback used as a regular callback
# notice the use of config
_ = model.fit(x_train, y_train,
          epochs=config.epochs,
          batch_size=config.batch_size,
          validation_data=(x_test, y_test),
          callbacks=[WandbCallback()])

```

If you are working on images and need a way to log your correctly classified and misclassified sample images, you can also do so using wandb. Below you will see the examples of it.

```python

# in order to get prediction on small subset of images.
val_images, val_labels = x_test[:32], y_test[:32]

# train with our favorite model.fit
# notice WandbCallback used as a regular callback
# notice that we are passing in some arguments as well
# notice the use of config
_ = model.fit(x_train, y_train,
          epochs=config.epochs,
          batch_size=config.batch_size,
          validation_data=(x_test, y_test),
          callbacks=[WandbCallback(data_type='image',
                                   training_data=(val_images, val_labels),
                                   labels=CLASS_NAMES)])
```

[![logging images with wandb](/images/wand-log-images.png)](https://ibb.co/ZcbvNP9)

## Log Custom metrics with wandb.log

You can always log custom metrics and extra information using wandb.log .

```python
loss, accuracy = model.evaluate(x_test, y_test)
print('Test Error Rate: ', round((1-accuracy)*100, 2))

# notice the use of wandb.log.
# We can easiy pass in values as key-value pairs.
wandb.log({'Test Error Rate': round((1-accuracy)*100, 2)})

```

# wandb Dashboard

[![Wandb dashboard](/images/wandb-dashboard.png)](https://ibb.co/tBZDJzW)

You can see all your logged metrics in your wandb project dashboard. Wandb Groups information into different sections like Overview, Charts ,logs, system, model and files.

## Overview

[![Wandb dashboard](/images/wandb-overview.png)](https://ibb.co/RvNFHTW)

The information in the `overview` section is pretty intuitive and self-explanatory. However, the Git Repository field and the Git State field are worthy of special mention. You can run the checkout command in the Git State field to pin down the exact code for reproducing the experiment. Under the hood, wandb tracks all the changes you made to the original repo, and save the "diff" files in a local directory. In this way, you can easily switch between different versions of the code without manually pushing the change to the remote repo.

## Logs

The `logs` section shows the console output during the experiment. This is useful for debugging the performance of the model.

[![Wandb dashboard](/images/wandb-logs.png)](https://ibb.co/hmKFC7H)

## Systems

To me, the system section is where wandb really shines and separates itself from other options such as TensorBoard. It is a centralized place to track system utilization during the experiment. There are in total of 8 graphs displayed in this section. These graphs give you insight into the training bottleneck and possible ways to uplift it. For example, below are the diagrams of the experiment:

[![Wandb dashboard](/images/wandb-system.png)](https://ibb.co/VNGQbyQ)

## Tensorboard

wandb also provide options for people who love tensorboard. You can directly sync wandb with tensorboard by just setting `sync_tensorboard = True` in your `wandb.init`. So, that every information that is logged into tensorboard will also be logged into wandb.

```python

wandb.init(
    name = '<your run name>',
    project = 'your project name',
    config = '<your config>',
    sync_tensorboard = True)
```

# Summary

We discuss how to integrate wandb in any deeplearning framework using python for inspecting the efficiency of training jobs. We also looked into other aspects of wandb that makes it so much unique than other training logging software like tensorboard.

To learn more about wandb, check out their website: [https://www.wandb.com/](https://www.wandb.com/).


<p>Wandb Tool for visualizing and tracking your machine learning experiments better than tensorboard.</p>

Wandb Your machine learning project.


## Table of contents

Speech processing is very first phase in any speech system either it is speech recognition system or speaker Diarization or something else. Speech processing plays an important role in speech system to extract vocal features i.e identify the components of the audio signal that are good for identifying the linguistic content and discarding all other stuff which carries information like background noise, emotion etc.

In this post we will learn very important mathematical concept about speech processing in any speech system and implement the mathematics in python.

Mel Frequency Cepstral Coefficents (MFCCs) and Filter Banks are a feature widely used in automatic speech and speaker recognition. But Filter Banks is more popluar Nowadays due to its robustness on mapping vocal features.Computing filter banks and MFCCs involve somewhat same procedure, where in both cases filter banks are computed and with a few more extra steps MFCCs can be obtained.

Let's get started with loading speech signal with python. I will be using python 3.6 for this post.

```python

import numpy as np
from scipy.io import wavfile
from scipy.fftpack import dct
from matplotlib import pyplot as plt

sample_rate, signal = wavfile.read('../audio/mfcc.wav')

signal = signal[0:int(10* sample_rate)]
Time = np.linspace(0, len(signal) / sample_rate, num=len(signal))

plt.plot(Time, signal)
```

The raw signal has the following from in the time domain:
[![raw signal image of speech](/images/spectral-image-speech.png)](https://ibb.co/nPvnTGc)

## Pre-Emphasis

The first step is to apply a pre-emphesis filter on signal to amplify the high frequencies. A pre-emphesis filter is useful in several ways:

- balance frequency spectrum since high frequencies usually have smaller magnitudes compared to lower frequencies
- avoid numerical problems during fourier operation
- Might also improve the signal to Noise Ratio (SNR)

The pre-emphesis filter can be applied to a signal x using the first order filter in the following equation:

$$ y(t) = x(t) - \alpha x(t-1) $$

which can be easily implemented using the following line, where typical values for the filter coefficeint ( $\alpha$ ) are 0.95 to 0.97,
`pre_empasis = 0.97`

```python

pre_emphasis = 0.97
emphasized_signal = np.append(signal[0], signal[1:] - pre_emphasis * signal[:-1])

```

Pre-emphasis has a modest effect in modern systems, mainly because most of the motivations for the pre-emphasis filter can be achieved using mean normalization except for avoiding the Fourier transfrom numerical issues which should not be a problem in modern FFT implementations.

The signal after pre-emphasis has the following effect in orignal signal.

[![before and after pre-emphesis](/images/before-after-preemphesis.png)](https://ibb.co/Jqn0x33)

## Framing

After pre-emphasis, we need to split the signal into short-time frames. The rationale behind this step is that frequencies in a signal change over time, so in most cases it dosen't make sense to do the Fourier Transform across the entier signal in that we would lose the frequency contours of the signal over time. To avoid that, we can safely assume that frequencies in a signal are stationary over a very short period of time. Therefore, by doing a Fourier transform over this short-time frame, we can obtain a good approximation of the frequency contours of the signal by concatenating adjacent frames.

Typical frame sizes in speech processing rnage from 20 ms to 40 ms with 50% (+/- 10%) overlap between conseutive frames.Popular settings are 25 ms for the frame size, frame_size = 0.025 and a 10 ms stride (15 ms overlap), frame_stride = 0.01

```python

frame_size = 0.025
frame_stride = 0.01

frame_length, frame_step = frame_size * sample_rate, frame_stride * sample_rate  # Convert from seconds to samples
signal_length = len(emphasized_signal)
frame_length = int(round(frame_length))
frame_step = int(round(frame_step))
num_frames = int(np.ceil(float(np.abs(signal_length - frame_length)) / frame_step))  # Make sure that we have at least 1 frame

pad_signal_length = num_frames * frame_step + frame_length
z = np.zeros((pad_signal_length - signal_length))
pad_signal = np.append(emphasized_signal, z) # Pad Signal to make sure that all frames have equal number of samples without truncating any samples from the original signal

indices = np.tile(np.arange(0, frame_length), (num_frames, 1)) + np.tile(np.arange(0, num_frames * frame_step, frame_step), (frame_length, 1)).T
frames = pad_signal[indices.astype(np.int32, copy=False)]

```

## Window

After slicing the signal into frames, we apply a window function such as the Hamming window to each frame. A Hamming window has the following form:
$$ w[n] = 0.54 − 0.46 cos ( \frac{2πn}{N − 1} ) $$
where,  0 $\leq$ n $\leq$ N-1,N is the window length.

There are several reasons why we need to apply a window function to the frames, notably to counteract the assumption made by the FFT that the data is infinite and to reduce spectral leakage.

````python

frames *= np.hamming(frame_length)```
# frames *= 0.54 - 0.46 * np.cos((2 * np.pi * n) / (frame_length - 1))  # Explicit Implementation **
````

## Fourier-Transfrom and Power Spectrum

We can now do an N-point FFT on each frame to calculate the frequency spectrum, which is also called Short-Time-Fourier-Transfrom (STFT), where N is typically 256 or 512, `NFFT = 512`; and then compute the power spectrum (Periodogram) using the following equation:
$$P = \frac{|FFT(x_i)|^2}{N}$$
where , $x_i$ is the $i^{th}$ frame of signal $x$. This can be easily imlemented with the following lines:

```python
NFFT = 512

mag_frames = np.absolute(np.fft.rfft(frames, NFFT))  # Magnitude of the FFT
pow_frames = ((1.0 / NFFT) * ((mag_frames) ** 2))  # Power Spectrum
```

## Filter Banks

The final step to computing filter banks is applying triangular filters, typically 40 filters, nfilt = 40 on a Mel-scale to the power spectrum to extract frequency bands. The Mel-Scale aims to mimic the non-linear human ear perception of sound, by being more discriminative at lower frequenceis and less discriminative at higher frequencies. We can convert between Hertz ($f$) and Mel ( $m$) using the following equations:
$$ \Large m = 2595 \log\_{10} (1 + \frac{f}{700}) \approx 1125 \ln (1 + \frac{f}{700} ) $$

$$ \Large f = 700 ( 10^{ \frac{m}{2595} } -1)$$

Each filter in the filter bank is triangular having a response of 1 at the center frequency and decrease linearly towards 0 till it reaches the center frequencies of the two adjacent filters where the response is 0. which can be modeled by the following equation:

$$
H_m(k) =
  \begin{cases}
      \ 0                                      & k < f(m - 1) \\
      \\
      \dfrac{k - f(m - 1)}{f(m) - f(m - 1)}  & f(m - 1) \leq k < f(m) \\
      \\
      1                                      & k = f(m) \\
      \\
      \dfrac{f(m + 1) - k}{f(m + 1) - f(m)}  & f(m) < k \leq f(m + 1) \\
      \\
      0                                      & k > f(m + 1) \\
  \end{cases}
$$

```python
nfilt = 40

low_freq_mel = 0
high_freq_mel = (2595 * np.log10(1 + (sample_rate / 2) / 700))  # Convert Hz to Mel
mel_points = np.linspace(low_freq_mel, high_freq_mel, nfilt + 2)  # Equally spaced in Mel scale
hz_points = (700 * (10**(mel_points / 2595) - 1))  # Convert Mel to Hz
bin = np.floor((NFFT + 1) * hz_points / sample_rate)

fbank = np.zeros((nfilt, int(np.floor(NFFT / 2 + 1))))
for m in range(1, nfilt + 1):
    f_m_minus = int(bin[m - 1])   # left
    f_m = int(bin[m])             # center
    f_m_plus = int(bin[m + 1])    # right

    for k in range(f_m_minus, f_m):
        fbank[m - 1, k] = (k - bin[m - 1]) / (bin[m] - bin[m - 1])
    for k in range(f_m, f_m_plus):
        fbank[m - 1, k] = (bin[m + 1] - k) / (bin[m + 1] - bin[m])
filter_banks = np.dot(pow_frames, fbank.T)
filter_banks = np.where(filter_banks == 0, np.finfo(float).eps, filter_banks)  # Numerical Stability
filter_banks = 20 * np.log10(filter_banks)  # dB
```

After applying the filter bank to the power spectrum (peridogram) of the signal, we obtain the following spectrogram:

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(20, 4))
cax = ax.matshow(
    np.transpose(filter_banks),
    interpolation="nearest",
    aspect="auto",
    cmap=plt.cm.afmhot_r,
    origin="lower",
)
fig.colorbar(cax)
plt.title("Mel compression Spectrogram")
plt.show()
```

[![melspectrogram of speech signal](/images/mfcc-spectorgram.png)](https://ibb.co/xSyfhsV)

## Mel-frequency cepstral Coecfficents (MFCCs)

It turns out that filter bank coefficients computed in the previous step are higly correlated, which could be problematic in some machine learning algorithms. Therefore, we can apply Discrete Cosine Transform(DCT) to decorrelate the filter bank coefficients and yield compressed representation of filter banks. Typically, for Automatic SPeech Recognition (ASR), the resulting cepstral coefficeints 2- 13 are retained and the rest are discareded; `num_ceps = 12`. The reasons for discarding the other coefficeints is that they represent fast changes in the filter bank coefficients and these fine details don't contribute to ASR.

```python
num_ceps = 12
mfcc = dct(filter_banks, type = 2, axis=1, norm="ortho")[:,1: (num_ceps + 1)] # keep 2-13

```

One may apply sinusoidal liftering to the MFCCs to de-emphasize higher MFCCs which has been claimed to improve speech recognition in noisy signals.

```python

cep_lifter = 22
(nframes, ncoeff) = mfcc.shape
n = np.arange(ncoeff)
lift = 1 + (cep_lifter / 2) * np.sin(np.pi * n/ cep_lifter)
mfcc *= lift

fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(20, 4))
cax = ax.matshow(
    np.transpose(mfcc),
    interpolation="nearest",
    aspect="auto",
    cmap=plt.cm.afmhot_r,
    origin="lower",
)
fig.colorbar(cax)
plt.title("MFCC Spectrogram")
plt.show()
```

[![MFCC spectrogram of speech signal](/images/mel-spectrogram.png)](https://ibb.co/S7c5pnL)

## Mean Normalization

As previously mentioned, to balance the spectrum and improve the Signal-to-Noise (SNR), we can simply substract the mean of each coefficeint from all frames,

```python
filter_banks -= (np.mean(filter_banks, axis = 0) + 1e-8)
```

and similarly for MFCCs:

```python

mfcc -= (np.mean(mfcc, axis = 0) + 1e-8)

```

To this point, the steps to compute filter banks and MFCCs were discussed in terms of their motivation and implementations.It is interesting to note that all steps needed to compute filter banks were motivated by the nature of the speech signal and the human perception of such signals.

On the Contrary, the extra steps needed to compute MFCCs were motivated by the limitation of some classifical machine learning algorithms.The Discrete cosine Transfrom (DCT) was needed to decorrelate filter banks coeffiicients, a process refer as whitening. In particular, MFCCs were very popular with Gaussian Mixture Models - Hidden Markov Models(GMM - HMM).
But with the advent of Deep learning in speech system, one might not need DCT.


<p>Signal Processing and Speech Recognition using python</p>

Speech Signal Processing using python


## Table of contents

Testing, by which we implicitly mean _automated testing_, is the practice of writing small programs that check that the code under test ( the production code ) behaved as expected for certain input pools, which is usually either carefully chosen to exercise certain features or randomized to ensure broad coverage.

Go's approach to testing seems rather low-tech in comparison to some big language like java, c++. It relies on one command, `go test`, and a set of conventions for writing test functions that go test can run. The comparatively lightweight mechanism is effective for pure testing, and it extends naturally to benchmarks and systematic example for documentation. The best thing about writing test code in Go is , that test code is no different from code if we intend to implement API we are testing.We focus on short functions that focus on one part of the task.We have to be careful about boundary conditions. think about data structures, and reason about what results a computation should produce from suitable inputs. But is the same as writing ordinary Go code.

# The GO test Tool

Testing in Go starts with `go test` subcommand i.e. test driver for Go packages that are organized according to certain conventions. `go test` command specifically looks and execute files that ends with **\_test.go** which are not part of the package when built by `go build `.

The Go test files contains three kinds of special functions :

- A _test function_: Function which name starts with `Test`. This function exercises some program logic for correct behaviour; subcommand `go test` calls the test function and reports the result which is either `PASS` or `FAIL`.

- A _benchmark function_: It has the name beginning with `Benchmark` and measures the performance of some operations; subcommand `go test` reports the mean execution time of the operation.

- An _example function_: Its name starts with `Example` provides machine-checked documentation.

The `go test` tool scans the \*\_test.go files for these special functions, generates a temporary main package that calls them all in the proper way, builds and runs it, reports the result, and then cleans up.

# Test Functions

Each test file must import the testing package. Test functions has following signature :

```go
func TestName(t *testing.T) {
  // ..
}

```

Test function names must begin with Test; The optional suffix Name must begin with a capital letter:

```go
func TestSin(t *testing.T) { // ... }
func TestCos(t *testing.T) { // ... }
```

The t parameter provides methods for reporting test failures and logging additional information. Lets create a new package palindrome containing a single function IsPalindrome that reports whether a string a string reads same forward and backward.

```go
package palindrome

func IsPalidrome(s string) bool {
  for i := range s {
    if s[i] != s[len(s) - 1 - i] {
      return false
    }
  }
  return true
}

```

In the same directory, the palindrome_test.go contains two test functions named TestPalindrome and TestNonPalindrome. Each check that IsPalindrome gives the right answer for a single input and reports failures using t.Errorf:

```go
package palindrome

func TestPalindrome(t *testing.T) {
  if !IsPalindrome("detartrated") {
    t.Errorf(`IsPalindrome("detartrated") = false`)
  }
  if !IsPalindrome("kayak") {
    t.Errorf(`IsPalindrome("kayak") = false`)
  }
}

func TestNonPalindrome(t *testing.T) {
  if IsPalindrome("palindrome") {
    t.Error(`IsPalindrome("palindrome") = true`)
  }
}

```

Now,we already have test code to test our functionality, we can use go test command to run tests in go.
Remember `go test palindrome` will run the package level test.For this to work you have to place your package in either `$GOROOT` or `$GOPATH` directory.
In our case we can test our functionlity by going into pacakge directory.

```bash
> cd <path_to_project_dir>/palindrome
> go test

PASS
ok  	<path_to_project_dir>/palindrome	0.001s
```

To look into more details on test function, To see which one failed and which one succeed, we can use `-v` flag , which will prints the name and execution time of each test in the package.

```bash
> go test -v

=== RUN TestPalindrome
--- PASS: TestPalindrome (0.00s)
=== RUN TestNonPalindrome
--- PASS: TestNonPalindrome (0.00s)
=== RUN TestFrenchPalindrome
--- FAIL: TestFrenchPalindrome (0.00s)
word_test.go:28: IsPalindrome("été") = false
=== RUN TestCanalPalindrome
--- FAIL: TestCanalPalindrome (0.00s)
word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false
FAIL
exit status 1
FAIL  <path_to_project_dir>/palindrome 0.001s

```

and the -run flag, whose argument is a regular expression, causes go test to run only those tests whose function name matches the patter:

```bash

> go test -v -run="French|Canal"
=== RUN TestFrenchPalindrome
--- FAIL: TestFrenchPalindrome (0.00s)
word_test.go:28: IsPalindrome("été") = false
=== RUN TestCanalPalindrome
--- FAIL: TestCanalPalindrome (0.00s)
word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false
FAIL
exit status 1
FAIL  <path_to_project_dir>/palindrome 0.001s

```

# Benchmark Function

Benchmarking is the practice of measuring the performance of a program on a fixed workload. In Go, a benchmark function look like a test function, but with the
Benchmark prefix and a *testing.B parameter that provides most of the same methods as a *testing.T, plus few extra related performance measurement.It also exposes
an Integer field N, which specifies the number of times to perform the operation being measured.

Here's a benchmark for IsPalindrome that calls it N times in a loop.

```go
func BenchmarkIsPalindrome(b *testing.B) {
  for i := 0; i < b.N; i++ {
    IsPalindrome("A man, a plan, a canal: Panama")
  }
}
```

Unlike tests, by default no benchmarks are run. The argument to the -bench flag selects which benchmark to run. It is regular expression matching the names of Benchmark functions, with a default value that matches none of the functions. The "." pattern causes it to match all benchmark functions in package.

```bash
> go test -bench=.
goos: linux
goarch: amd64
BenchmarkIsPalindrome-12    	 4377900	       301 ns/op
PASS
ok  	<path_to_project_dir>/palindrome	1.598s
```

The benchmark name’s numeric suffix, 12 here , indicates the value of GOMAXPROCS, which is important for concurrent benchmarks.

The report tells us that each call to _IsPalindrome_ took about 0.301 microseconds, averaged over 4377900 runs. Since the benchmark runner initially has no idea how long the operation takes, it make some initial measuremetns using small values of N and then extrapolates to a value large enough for a stable timing measurement to be made.


<p>Testing Framework in Go is extremely simple and minimal</p>

/Aaditya.

MY WRITINGS

Blog Posts