CI/CD for AKS apps with Azure Pipelines

21 July 2024 Administrator Services, Solutions, Technology

Dataflow

A pull request (PR) to Azure Repos Git triggers a PR pipeline. This pipeline runs fast quality checks such as linting, building, and unit testing the code. If any of the checks fail, the PR doesn't merge. The result of a successful run of this pipeline is a successful merge of the PR.
A merge to Azure Repos Git triggers a CI pipeline. This pipeline runs the same tasks as the PR pipeline with some important additions. The CI pipeline runs integration tests. These tests require secrets, so this pipeline gets those secrets from Azure Key Vault.
The result of a successful run of this pipeline is the creation and publishing of a container image in a non-production Azure Container Registry.
The completion of the CI pipeline triggers the CD pipeline.
The CD pipeline deploys a YAML template to the staging AKS environment. The template specifies the container image from the non-production environment. The pipeline then performs acceptance tests against the staging environment to validate the deployment. A manual validation task is run if the tests succeed, requiring a person to validate the deployment and resume the pipeline. The manual validation step is optional. Some organizations will automatically deploy.
If the manual intervention is resumed, the CD pipeline promotes the image from the non-production Azure Container Registry to the production registry.
The CD pipeline deploys a YAML template to the production AKS environment. The template specifies the container image from the production environment.
Container Insights periodically forwards performance metrics, inventory data, and health state information from container hosts and containers to Azure Monitor.
Azure Monitor collects observability data such as logs and metrics so that an operator can analyze health, performance, and usage data. Application Insights collects all application-specific monitoring data, such as traces. Azure Log Analytics is used to store all that data.

Ref: https://learn.microsoft.com/en-us/azure/architecture/guide/aks/aks-cicd-azure-pipelines

How ChatGPT Works: The Model Behind The Bot

20 February 2023 Administrator

Molly Ruby

1.2K Followers

Exploring the world through data

A brief introduction to the intuition and methodology behind the chat bot you can’t stop hearing about.

This gentle introduction to the machine learning models that power ChatGPT, will start at the introduction of Large Language Models, dive into the revolutionary self-attention mechanism that enabled GPT-3 to be trained, and then burrow into Reinforcement Learning From Human Feedback, the novel technique that made ChatGPT exceptional.

Large Language Models

ChatGPT is an extrapolation of a class of machine learning Natural Language Processing models known as Large Language Model (LLMs). LLMs digest huge quantities of text data and infer relationships between words within the text. These models have grown over the last few years as we’ve seen advancements in computational power. LLMs increase their capability as the size of their input datasets and parameter space increase.

The most basic training of language models involves predicting a word in a sequence of words. Most commonly, this is observed as either next-token-prediction and masked-language-modeling.

Arbitrary example of next-token-prediction and masked-language-modeling generated by the author.

In this basic sequencing technique, often deployed through a Long-Short-Term-Memory (LSTM) model, the model is filling in the blank with the most statistically probable word given the surrounding context. There are two major limitations with this sequential modeling structure.

The model is unable to value some of the surrounding words more than others. In the above example, while ‘reading’ may most often associate with ‘hates’, in the database ‘Jacob’ may be such an avid reader that the model should give more weight to ‘Jacob’ than to ‘reading’ and choose ‘love’ instead of ‘hates’.
The input data is processed individually and sequentially rather than as a whole corpus. This means that when an LSTM is trained, the window of context is fixed, extending only beyond an individual input for several steps in the sequence. This limits the complexity of the relationships between words and the meanings that can be derived.

In response to this issue, in 2017 a team at Google Brain introduced transformers. Unlike LSTMs, transformers can process all input data simultaneously. Using a self-attention mechanism, the model can give varying weight to different parts of the input data in relation to any position of the language sequence. This feature enabled massive improvements in infusing meaning into LLMs and enables processing of significantly larger datasets.

GPT and Self-Attention

Generative Pre-training Transformer (GPT) models were first launched in 2018 by openAI as GPT-1. The models continued to evolve over 2019 with GPT-2, 2020 with GPT-3, and most recently in 2022 with InstructGPT and ChatGPT. Prior to integrating human feedback into the system, the greatest advancement in the GPT model evolution was driven by achievements in computational efficiency, which enabled GPT-3 to be trained on significantly more data than GPT-2, giving it a more diverse knowledge base and the capability to perform a wider range of tasks.

Comparison of GPT-2 (left) and GPT-3 (right). Generated by the author.

All GPT models leverage the transformer architecture, which means they have an encoder to process the input sequence and a decoder to generate the output sequence. Both the encoder and decoder have a multi-head self-attention mechanism that allows the model to differentially weight parts of the sequence to infer meaning and context. In addition, the encoder leverages masked-language-modeling to understand the relationship between words and produce more comprehensible responses.

The self-attention mechanism that drives GPT works by converting tokens (pieces of text, which can be a word, sentence, or other grouping of text) into vectors that represent the importance of the token in the input sequence. To do this, the model,

Creates a query, key, and value vector for each token in the input sequence.
Calculates the similarity between the query vector from step one and the key vector of every other token by taking the dot product of the two vectors.
Generates normalized weights by feeding the output of step 2 into a softmax function.
Generates a final vector, representing the importance of the token within the sequence by multiplying the weights generated in step 3 by the value vectors of each token.

The ‘multi-head’ attention mechanism that GPT uses is an evolution of self-attention. Rather than performing steps 1–4 once, in parallel the model iterates this mechanism several times, each time generating a new linear projection of the query, key, and value vectors. By expanding self-attention in this way, the model is capable of grasping sub-meanings and more complex relationships within the input data.

Screenshot from ChatGPT generated by the author.

Although GPT-3 introduced remarkable advancements in natural language processing, it is limited in its ability to align with user intentions. For example, GPT-3 may produce outputs that

Lack of helpfulness meaning they do not follow the user’s explicit instructions.
Contain hallucinations that reflect non-existing or incorrect facts.
Lack interpretability making it difficult for humans to understand how the model arrived at a particular decision or prediction.
Include toxic or biased content that is harmful or offensive and spreads misinformation.

Innovative training methodologies were introduced in ChatGPT to counteract some of these inherent issues of standard LLMs.

ChatGPT

ChatGPT is a spinoff of InstructGPT, which introduced a novel approach to incorporating human feedback into the training process to better align the model outputs with user intent. Reinforcement Learning from Human Feedback (RLHF) is described in depth in openAI’s 2022 paper Training language models to follow instructions with human feedback and is simplified below.

Step 1: Supervised Fine Tuning (SFT) Model

The first development involved fine-tuning the GPT-3 model by hiring 40 contractors to create a supervised training dataset, in which the input has a known output for the model to learn from. Inputs, or prompts, were collected from actual user entries into the Open API. The labelers then wrote an appropriate response to the prompt thus creating a known output for each input. The GPT-3 model was then fine-tuned using this new, supervised dataset, to create GPT-3.5, also called the SFT model.

In order to maximize diversity in the prompts dataset, only 200 prompts could come from any given user ID and any prompts that shared long common prefixes were removed. Finally, all prompts containing personally identifiable information (PII) were removed.

After aggregating prompts from OpenAI API, labelers were also asked to create sample prompts to fill-out categories in which there was only minimal real sample data. The categories of interest included

Plain prompts: any arbitrary ask.
Few-shot prompts: instructions that contain multiple query/response pairs.
User-based prompts: correspond to a specific use-case that was requested for the OpenAI API.

When generating responses, labelers were asked to do their best to infer what the instruction from the user was. The paper describes the main three ways that prompts request information.

Direct: “Tell me about…”
Few-shot: Given these two examples of a story, write another story about the same topic.
Continuation: Given the start of a story, finish it.

The compilation of prompts from the OpenAI API and hand-written by labelers resulted in 13,000 input / output samples to leverage for the supervised model.

Image (left) inserted from Training language models to follow instructions with human feedback *OpenAI et al., 2022* https://arxiv.org/pdf/2203.02155.pdf. Additional context added in red (right) by the author.

Step 2: Reward Model

After the SFT model is trained in step 1, the model generates better aligned responses to user prompts. The next refinement comes in the form of training a reward model in which a model input is a series of prompts and responses, and the output is a scaler value, called a reward. The reward model is required in order to leverage Reinforcement Learning in which a model learns to produce outputs to maximize its reward (see step 3).

To train the reward model, labelers are presented with 4 to 9 SFT model outputs for a single input prompt. They are asked to rank these outputs from best to worst, creating combinations of output ranking as follows.

Example of response ranking combinations. Generated by the author.

Including each combination in the model as a separate datapoint led to overfitting (failure to extrapolate beyond seen data). To solve, the model was built leveraging each group of rankings as a single batch datapoint.

Step 3: Reinforcement Learning Model

In the final stage, the model is presented with a random prompt and returns a response. The response is generated using the ‘policy’ that the model has learned in step 2. The policy represents a strategy that the machine has learned to use to achieve its goal; in this case, maximizing its reward. Based on the reward model developed in step 2, a scaler reward value is then determined for the prompt and response pair. The reward then feeds back into the model to evolve the policy.

In 2017, Schulman et al. introduced Proximal Policy Optimization (PPO), the methodology that is used in updating the model’s policy as each response is generated. PPO incorporates a per-token Kullback–Leibler (KL) penalty from the SFT model. The KL divergence measures the similarity of two distribution functions and penalizes extreme distances. In this case, using a KL penalty reduces the distance that the responses can be from the SFT model outputs trained in step 1 to avoid over-optimizing the reward model and deviating too drastically from the human intention dataset.

Steps 2 and 3 of the process can be iterated through repeatedly though in practice this has not been done extensively.

Evaluation of the Model

Evaluation of the model is performed by setting aside a test set during training that the model has not seen. On the test set, a series of evaluations are conducted to determine if the model is better aligned than its predecessor, GPT-3.

Helpfulness: the model’s ability to infer and follow user instructions. Labelers preferred outputs from InstructGPT over GPT-3 85 ± 3% of the time.

Truthfulness: the model’s tendency for hallucinations. The PPO model produced outputs that showed minor increases in truthfulness and informativeness when assessed using the TruthfulQA dataset.

Harmlessness: the model’s ability to avoid inappropriate, derogatory, and denigrating content. Harmlessness was tested using the RealToxicityPrompts dataset. The test was performed under three conditions.

Instructed to provide respectful responses: resulted in a significant decrease in toxic responses.
Instructed to provide responses, without any setting for respectfulness: no significant change in toxicity.
Instructed to provide toxic response: responses were in fact significantly more toxic than the GPT-3 model.

For more information on the methodologies used in creating ChatGPT and InstructGPT, read the original paper published by OpenAI Training language models to follow instructions with human feedback, 2022 https://arxiv.org/pdf/2203.02155.pdf.

Happy learning!

Sources

How to Create .NET Bot (DIY #2)

16 September 2022 Administrator

Hello there! Are you interested in creating your own .NET bot? I hope you are! This is the second part of a blog post telling you everything about Microsoft bots. In case you missed part 1, follow the link.

If you have already read the first part and you’ve got a wonderful little .NET bot at your disposal, let’s continue. First of all, thanks for reading on. Nothing feels better than creating a useful blog post. Second of all, let me show what you’re about to read. Three steps for going from a basic bot to a true AI machine:

Bringing Your .NET Bot to life
Connecting the bot to channels
Giving your Bot a voice

BRINGING YOUR .NET BOT TO LIFE

As mentioned before, you’re not starting from a blank page here. You already have a pretty sophisticated bot that uses LUIS service for brains. Now is the time to show your bot to the world. You need to deploy it, and what’s more important, test it beforehand.

For testing purposes, it’s better to use ngrok or any similar tool that secures tunnels to your localhost. Oh, and don’t be alarmed when your bot is a little bit ‘slow’ during your tests. The time lag when your bot is answering your questions will not be present when deployed.

When you get tired testing, you’ll know it’s time—time to deploy the bot! There are two ways of dealing with the deployment:

1. Use Azure—this is an out-of-the-box solution from Microsoft. It has a free trial available for a month. Furthermore, it’s not that expensive… Only about 10 dollars a month for a basic hosting of your one and only bot. However, if you need rough numbers, Microsoft has a wonderful calculator (link).

2. Do it yourself (DIY)—this one is always an option. To make it work, you need 3 things:

Your own server or an affordable third-party vendor
To make sure your bot has an Internet-accessible endpoint
A valid HTTPS certificate available

You’re almost set and done. Still, before we move on, you need to register your bot in the Bot Framework Portal. Small tip: when you generate Microsoft App credentials, make sure you’ve copied and securely stored the app password. There’s no way of restoring your lost password here. So, you may end up losing the whole Bot Framework account. And you don’t want that!

CONNECTING THE BOT TO CHANNELS

Finally, your bot is alive and breathing. I guess, he should start doing his job—interacting with people. There are a dozen places where you can publish it: Bing, Cortana, Messenger, Skype, etc. Two choices are pre-configured for you—Skype and Web Chat. However, if you don’t find a channel here that suits you, there’s always a custom API available.

Well, let’s go through all the available channels:

1. Skype—It’s a default channel providing only simple one-to-one messaging from the start. Nevertheless, you can configure web control, group chatting and calling. As for the calls, you have two choices: Interactive voice response—This option is limited to audio. Moreover, there’s no group call option for now. Real-time media—This option has the chat and the group call options. However, it’s not interactive at any level. In order to use Real-time media capability, you need to download and apply Bot.Builder.RealTimeMediaCalling SDK and Microsoft.Skype.Bots.Media lib.

2. Web-chat—It’s also a default channel that has only a simple chatting option. In Microsoft Bot Framework, you can generate a piece of code that you can add to almost any website. Simple and useful.

3. Facebook Messenger—It requires a Facebook company page, and I wonder whether there’s a business in the world without one. On your FB page, you should enable your Messenger app and grab all the configurations for your channel.

4. Telegram—This one’s simple! You have to register your bot with@BotFather.

5. Bing—Users can interact with your bot directly in the Bing search results page. This is very healthy for your general SEO strategy and any keywords that you might want to associate your bot with.

6. Cortana—Your bot can be added to Cortana skills. However, bear in mind that your bot should be designed both for speech and text in order to be connected with Cortana.

7. Email—Connecting to this old school channel can also prove useful. However, your .NET bot can only communicate with users via Office 365.

8. Additional Channels—GroupMe, Kik, Microsoft Teams & Twilio. Sorry, I’m not giving you a description of these channels for a simple reason. As the author of the article, I’ve never used any of them, and I’m no authority on the subject.

9. Direct Line (Custom API)—This is basically a REST API you can use to build a channel for any communication platform that’s not already available. You can find detailed features of the Direct API in this link.

It’s your marketing genius that should decide which channels to use and which to ignore. Know where your target audience lives. Create a bot specifically for them and their needs, and you’re one step closer to success. Finally, your bot will bring you value, and maybe even some profit.

GIVING YOUR BOT A VOICE

You have a pretty sophisticated bot on your hands that’s getting smarter and smarter with every conversation. However, at this point, he’s only good for a silent chatbox. No worries, we can make him talk! We can make him talk with the help of the Bot Builder Calling SDK. Step by step, setting up each and every element of the SDK.

For the detailed instructions, please follow the link to a wonderful @ankitbko blog post: Skype Call your bot—Microsoft Bot Framework with Bing Speech.

All in all, I hope you enjoyed reading or skimming through, whatever you prefer! Ideally, at the end of the blog post, you should come up with a bot like this one: Trivia Chatbot.

How to Create .NET Bot (DIY #1)

16 September 2022 Administrator

Oleg Romanyuk

You’ve heard a lot of buzz about bots, artificial intelligence, and machine learning. Maybe, you’ve wondered how practical a . NET bot is and should you adopt the new, trendy tech. All in all, one thing you should know about bots and all the tech that empowers them. Bot technologies are here to stay! The main reason: they are cheaper than your local working guy. If you need the proof, check out our wonderful blog post — AI Bots are Coming for Your Testing Jobs or just browse the web.

Bots will save you money. And despite all the hi-tech chatter in the media, they’re not that complicated to make. We will tell you how to create your own .NET bot in 6 easy steps and 2 short blog posts:

DIY (Part 1)

Introduction to the Bot Framework for .NET
Create Your First .NET Bot
Make the Bot Smarter

DIY (Part 2)

Bring Your Bot to Life
Connect the Bot to Channels
Give Your Bot a Voice

Introduction to the Bot Framework for .NET

Let’s make everything clear from the start. When we use the word ‘bot’, we mean an app that users interact with in a conversational way. You can always create a proper bot from scratch but let’s leave it to digital cavemen. There’s a framework for everything software nowadays, especially bots. Moreover, the .NET bot framework will solve some of your basic issues by providing:

Basic I/O (input-output processing)
Language and dialogue skills
Connection to users
Language support

The .NET bot framework itself consists of the following tools and features:

Bot Builder (with SDKs for .NET)
Bot Framework Emulator
Developer Portal
Bot Connector Service
Channel APIs

How it all works? So simple you won’t believe it. First, you’ve got your Bot Builder SDK for actual coding together with the Developer Portal for additional services like APIs, databases, Azure, machine learning etc. Additionally, there’s a Bot Framework Emulator for testing your code.

Second, you have a Bot Connector Service for managing the input-output messaging

Third, there’s a whole collection of Channel APIs, including social media. If you want to add the bot to the channel that’s not provided (image below), remember there’s a custom API builder. In other words, you’ve got everything you need for your first .NET bot.

Create Your First .NET Bot

Okay, enough with the theory! Let’s get to the business. Follow the steps:

Step 1. Install Bot Builder SDK via Nuget Package Manager

Step 2. Install the Bot Application template to your Visual Studio

Step 3. Use the source code from GitHub to get a ready-made bot

Step 4. Use the Bot Framework Emulator, in order to test out your bot

The final step will be securing your newly built bot. This will require a few additional measures. However, securing a good piece of software always pays back.

There are several rules how to secure your bot properly:

use only an HTTPS endpoint
Enable a Bot Framework authentication by registering your bot to acquire a Microsoft App ID and a password
specify its app ID and password in your bot’s web.config file (look for the tip)
use the [BotAuthentication] attribute to specify authentication credentials when using Bot Builder SDK for .NET

Tip: The ID authentication goes like this. You add the Microsoft App ID in a proper field. Wait for the initial authentication. Add the newly generated BotID to a proper field and to the config(Bot handle). That’s it!

Still puzzled? Click here for a Microsoft tutorial on how to build your bot.

Moreover, in its Bot Framework, Microsoft offers a wonderful glossary of all the concepts that can be unfamiliar: https://docs.microsoft.com/en-us/bot-framework/dotnet/bot-builder-dotnet-concepts.

Hopefully, at the end of the chapter, you’re all set and done with creating your first bot. In case it’s not the case, please address me — the author of the article — Oleh Romanyuk @ Linkedin here. I will make sure you have a proper .NET bot created!

Make Your Bot Smarter

Congratulations — you’ve made it this far. Your bot is alive and you are a proud botfather. However, the bot is still primitive and pretty much useless. At this point, Microsoft provides a whole bunch of solution for boosting the brains of your bot, specifically 5 different APIs for language understanding.

Don’t know which to choose? Don’t panic. The first one is the most useful one. The name is LUIS. LUIS (Language Understanding Intelligent Service) is able to process natural language using pre-built or custom-trained language models. It has a bunch of advantages that will suit you just fine (image below).

Without any further introduction, let’s get to the drawing board and plan your LUIS app. What is the purpose of your bot and your business? How do you see your bot helping your clients?

For a smart plan for teaching the bot, let’s go through the following steps:

Step 1. Identify your domain — The way how LUIS functions is usually built around a domain-specific topic. For example, you have an app for selling tickets or an app for tracking your fitness. Both of them will be apps in different domains, and they will require different LUIS settings. In addition, there are 20 pre-built domains for your convenience that can be found here.

Step 2. Identify your intents — Intents are keywords that explain what your customer wants. If your customer writes ‘buy tickets’ obviously that’s what (s)he wants. If a fitness bot sees the word ‘pulse’ anywhere in the conversation, most probably the client needs to know something about his or her pulse. Plain and simple, make a list of all the keywords your customers can use in a conversation with your bot and connect them with an appropriate action from the side of the bot.

Step 3. Identify your entities — In order to book your flight, you need some information like the destination, date, airline, ticket category and travel class — that’s entities. You must add them in order to truly accomplish an intent.

So, here’s where all the fun starts. There are many types of entities you can create:

Simple entities — A simple entity describes a single concept.

Hierarchical entities — A hierarchical entity represents a category and its members.

Composite entities — A composite entity is made up of other entities that form parts of a whole.

Prebuilt entities — LUIS provides pre-built entities for common types like Number, which you can use for the number of tickets in a ticket order.

List entities — A list entity is an explicitly specified list of values. Each value consists of one or more synonyms. In a travel app, you might choose to create a list entity to represent airport names.

lanned your LUIS App? Sure what intents and entities you’ll use? Perfect! And no worries, if you’re not sure about everything, you’ll be able to change the settings afterward. Now’s the time to create an actual LUIS app and make your bot smarter!

WHAT IS SDLC?

15 September 2022 Administrator

VIKTORIA RUBAN

BA at Computools

The development of technology has shown people that successful products are supposed to make life easier and more comfortable. Get instant access to applications for ordering and delivering, booking and planning, online banking, a personal assistant, and so much more by simply clicking an icon on your smartphone. Building powerful software requires an integrated approach and careful planning. It’s time to find out more about SDLC.

WHAT IS SDLC?

SDLC stands for Software Development Life Cycle.

It’s a list of project phases that helps a team understand what actually needs to be done at each step, and what artifacts need to be ready to start a particular phase.

LET’S TAKE A LOOK:

SDLC table

1. IDEATION

At this stage, the team has a lot to learn about the client’s business and ideas.

Here are the basic actions conducted during the ideation phase:

● review the documentation sent by the client (if it exists);

● research the domain and spend time getting familiar with it (especially if it’s niche);

● conduct meetings with the stakeholders to discuss basic requirements, provide suggestions, and brainstorm ideas;

● make sure to consider the following:

○ Whose opinion can influence the product (it can be someone from the client’s side, specific departments like Marketing, or national or international regulations, laws, and institutions)?

○ What are the goals of these people, departments, or institutions?

○ Who is the product being made for?

○ Who will use the product?

○ Why is the product interesting and/or useful for these people?

○ Where do they live?

○ Do their daily routines need to be considered?

○ Does the client have any ideas about the monetization strategy for the product?

By the end of this first stage, the team already has some information about the product and found a common ground with the client.

○ general idea of the project;

○ high-level mind map;

○ stakeholders and their goals;

○ competitors;

○ key terms and definition

2. REQUIREMENTS

This stage is still a part of the discovery phase. It gives a business analyst a chance to gather any specific requirements for the project and prepare the documentation needed for the next step.

Below are the main techniques used to elicit requirements:

● interviewing;

● holding workshops;

● brainstorming;

● observing.

By the end of this stage, a software requirements specification (SRS) is drafted and continuously reviewed by a quality assurance specialist to help all technical and non-technical stakeholders stay on the same page. The document describes functional and non-functional requirements and comes with a prototype.

3. DESIGN

This step is based on work done during the previous step. A solution architect prepares documentation pertaining to software architecture describing the architectural approach and data flow for the particular project, as well as technical decisions and third-party integrations.

During this phase, a test strategy is also defined for the further stages.

The perfect case is when all the documentation can be checked by stakeholders in terms of product relevance and legal constraints.

By the end of this stage, the discovery phase of the software development life cycle is considered to be finished and the project is ready for development.

4. DEVELOPMENT

Only at this step developers start implementing the project according to prepared documentation.
The quality of this step greatly depends on developers’ experience and how well documents are detailed.

5. TESTING

This stage usually goes hand in hand with development. Testing provides quality assurance and makes sure that the project is implemented according to the requirements.

During testing, if any number of bugs are found, the team returns to development again. It’s common that a short cycle of these two steps will repeat until all bugs are removed.

There are several types of testing conducted at this stage:

● manual testing according to acceptance criteria;

● regression testing;

● load testing;

● unit-testing (optional);

● integrational (optional);

● auto testing (optional).

6. DEPLOYMENT

This is the final stage of the project. It implies deployment to the product server and/or digital distribution platforms.

7. MAINTENANCE

After product release, additional support may be needed. Additional documentation and support hours are discussed individually.

Several SDLC models can be used during the software development process:

● waterfall model;

● iterative model;

● V-model;

● spiral model;

● big bang model.

Each model has a specific set of steps and special processes. The team can experience different kinds of work efficiency by deploying different models. The most common is the iterative model.

Summing up, choosing an SDLC model suitable for your business and following the above mentioned software development life cycle steps helps facilitate work efficiency and successful product development.

SEEING A BIG PICTURE IS VITAL WHEN YOU BEGIN DEVELOPING SOFTWARE. SO BEFORE YOU START CRAFTING YOUR PRODUCT, TAKE A LOOK AT THE MOST COMMON AND SECURE SOFTWARE ENGINEERING LIFE CYCLE TO GRASP THE CONCEPT OF HOW A COMPLETE PROCESS LOOKS LIKE.

15 September 2022 Administrator

9 out of 10 startups fail because their solution has no market fit. This indicates that the software development analysis stage was skipped or conducted wrong. In such a case all further design, coding, and testing have little sense.

Seeing the big picture is vital when you begin developing software. So before you start crafting your product, take a look at the most common and secure software engineering life cycle to grasp the concept of how a complete process looks like.

1. ANALYSIS

Goal: To gather requirements and define the direction of the software engineering process.

Mistakes made during the analysis phase are the most expensive to fix later. This is the only time to build a foundation for your future product:

Carry out user research;
Analyze the competitors;
Establish business goals;
Measure success;
Map the customer journey;
Plan the budget.

It’s important to take into account the team on the client’s side. Its in-depth knowledge about the product goals and requirements, level of industry knowledge, and readiness to collaborate will contribute to the overall strategy and will make the discovery stage more effective. It’s crucial to engage at least one person from the client’s side to get a better understanding of the scope and goals of your product. Such interaction will help to write a detailed Software Requirements Specification (SRS) and to eliminate risks such as understaffing, over budget, and lack of market demand.

Outcome: Setting requirements and writing SRS

2. PRODUCT DESIGN

Goal: To convert requirements into detailed software architecture.

Product design can be divided into three fundamental components:

Functionality;
Appearance;
Quality.

Functionality is the priority of an Architect. By narrowing the focus he helps business and technical teams work together on a product that targets both clients’ needs and business goals.

Appearance is a tangible input of UX designers that conduct user research followed by sketching (any kind of it), prototyping, and creating MVP.

Quality stands for meeting customer needs and expectations with a product.

All the above components should be sourced from the SRS document that is a reference point for everyone involved in the project. The vital thing here is a vast understanding of the context for the existence of software engineering services. The more detailed the description of how the application needs to be created would be, the less additional input developers will require at the next coding stage.

Outcome: Software design description.

3. SOFTWARE DEVELOPMENT

Goal: Translate the design of the system into code.

This is a lengthy phase though less complicated than the previous two. Using the design description, programmers code the modules using the chosen programming language. The coding tasks are divided between the software engineering team members according to their skillset. Front-end developers create codes for displaying an application or product UI and the elements users need to interplay with the site. Their Back-end counterparts are in charge of the technical side of a product.

Establish conditions for organized and consistent coding:

Use proper guidelines;
Supervise every developer;
Automate deployment process;
Nourish the best programming practices.

Well-written code significantly reduces the number of test runs and maintenance-related problems at the next stages. At Erbis we use innovative design and development practices to drive our client’s growth.

Outcome: testable, functional software, and a Source Code Document.

4. PRODUCT TESTING

Goal: Code verification and bugs detection.

There is one step missing before mass-production testing. It consists of Quality Assurance (QA), Quality Control (QC), and Testing. While these are technically separate parts of the development process dependent on project size, stakeholders often group them as they have the same end goal: to make a high-quality product.

Let’s figure out the difference:

Quality Assurance. This is a process-oriented activity aimed to ensure that the team manages and creates deliverables consistently. The role of QA automation is to identify the reason for the error and to re-engineer the system so that such defects won’t appear any further. Thus focusing efforts on improving QA processes is one of the best investments an organization can make.
- Quality Control. This is a product-oriented performance related to intermediate and final results of software engineering. Examples of QC include technical reviews, software testing, and code inspections.
  - Testing. The tester creates tests and observes the behavior of the particular program under certain conditions. He fills in the documentation and returns it to developers. The best way to ensure that tests are run regularly is to automate them.

Outcome: Software is completely free of bugs and compliant.

5. DEPLOYMENT

Goal: Software delivery to a target device.

Software deployment refers to the process of running a product on a server or device and can be summarized in three general phases:

Preparation;
Testing;
Deployment.

A piece of software may be deployed several times during the software development lifecycle depending on its functioning and error check results. If it runs smoothly and the way it was intended, then consider your software ready to be launched for beta testing. The support team collects feedback from the first users, and if any bugs come up, the development team fixes them. After that, the final version is rolled out.

Outcome: Fully operational software in a live environment.

6 phases of the software development life cycle

6. MAINTENANCE

Goal: Ongoing security monitoring and update.

The process of software development is a never-ending cycle as the plan rarely turns out perfect when it meets reality. In most cases, product maintenance is the continuous phase intended to keep the software stable and up to date. If any new bugs and vulnerabilities appear, the maintenance team will mitigate them or flag them to be addressed in a future version of the software.

There are 2 types of maintenance:

Corrective. This means the fixation of defects that are rooted in production. They emerge because removing all the faults before delivery is extremely difficult.
Adaptive. It’s an addition of requirements you didn’t have in the original plan. Such modification takes shape due to environment or input data change.

Outcome: Utter user experience and productivity.

CONCLUSION

These steps are roughly the same from one software development life cycle model to another. They tend to occur in this order, though they can also be mixed into a rapidly-repeating cycle (like Agile) or break down into linear sequential phases (like Waterfall). Regardless of the method, the desired result of software engineering services is a competitive and customer-oriented product.

Reference: https://erbis.com/blog/6-phases-of-the-software-development-life-cycle/

What is Common Data Service (CDS) and why it is important for you if you use Power Apps or Power BI?

13 February 2022 Administrator

What is Common Data Service?

Common Data Service, abbreviated as CDS, is a data storage service. Like a database. You can use CDS to store data in the form of tables, which is called as Entities. Common Data Service is a service that is used mainly in the Power Apps portal, however, it is accessible through other Power Platform services and Microsoft Dynamics. the data can be loaded into CDS entities through multiple ways, and it can be also extracted from there through different methods. So you can say it this way that; CDS is a data storage and retrieval system, like a database.

Common Data Service (CDS) is a data storage system, like a database.

CDS includes a set of base entities (tables), but you can add custom entities to it. You can access CDS through other Power Platform services (Power BI, Power Apps, Power Automate…) and some other Microsoft services.

Why CDS is important for you if you are using Power Apps?

If you are using Power Apps, then it means you are creating a mobile application. The mobile application is most likely to work with data, capture information from the user through a data entry form or something and it needs to store the data somewhere. You would need to have a database system that you can store your data and retrieve it.

CDS is free storage for you in the Power Apps environment because you are already paying for the Power Apps license, then you can use CDS for free.

Of course, you can go and build your database in other systems, such as Azure SQL database, but then you need to pay for that service separately, or you might prefer to keep it on-prem in a SQL database, which then you would need to set up a gateway to use it. The choice of which database to use for your Power Apps app is up to you. However, CDS will give you a free, and easy-to-use database system to work with, and build your apps using that.

CDS is the free database service that you can use in Power Apps to store and retrieve the data of your apps.

So in a nutshell; CDS stores your Power Apps data at no extra cost, it is easy to manage. You don’t need a database developer to go and build a database for you to load your data into it. It is a data storage system that can be used by a citizen application developer.

You don’t need to know about databases, or be a database developer, to use CDS. It is built for the citizen app developer.

What the CDS database management system looks like?

Like many other database systems, CDS also has a management tool, which you can go and see entities, edit them and manage them. At the moment, you can use Power Apps portal for this management portal, and you will find CDS under the Power Apps portal like below;

What is the point of CDS if you are using Power BI?

There are two aspects of using CDS if you use Power BI. One is to use CDS as a data source system.

In the world of Power BI, we don’t store the data. We do, however, get data from a data storage system to analyze it. And that is why most of the people in the Power BI world, might not be familiar with CDS, because, from their point of view, this is just another database system, just another data source to get data from.

CDS is another data source that you can use when you Get Data in Power BI.

Another aspect of using CDS in Power BI is to use it as intermediate storage. Intermediate storage for your Power Query transformations. I have explained previously that why you might need to decouple your Power Query transformation layer into Power BI dataflows. Using CDS, you can store the output of dataflows into CDS, like a database, or let’s say, like a data warehouse, and use it for further analysis.

CDS can be your data warehouse if you use dataflows.

I highly recommend you to reach the article I wrote about decoupling the data transformation layer, data modelling layer, and visualization layer in Power BI implementation, which explains how dataflows can be an essential part of implementation for multi-developer architecture;

And the concept of dataflow is nowadays, not just for Power BI, but also for Power Apps. It is Power Platform Dataflows;

What is the storage engine behind the scene for CDS?

CDS stores the data, retrieve it and controls it using Azure services. There are a number of Azure services that are involved with this. Azure SQL DB, SQL elastic pools used for relational data, Blob storage for the non-relational data, and CosmosDB for logs. the screenshot below from Ryan Jones session at Microsoft Ignite 2019 explains how things placed together:

Ways to Load data into CDS

Because CDS is a storage system, you might ask how you can store data into it? what are ways? here is the answer:

Power Apps app. You can build an app using Power Apps that store the data into entities of CDS.
Power Apps portal using Get Data and Power Query experience.
Dataflow: In Power Apps portal, you can create a dataflow and schedule it to load data into CDS.
Other services
…

Way to retrieve data from CDS

You can extract data from CDS in many different ways, including:

Power Apps app; You can have forms in your app that show the existing data from entities of CDS.
Export data from Power Apps portal
Dataflow
Power BI, Get data from Common Data Services
Other services
…

How much does it cost for you?

If you are using Power Apps, then you have a license that covers also CDS, so you don’t need to pay anything extra. However, different licenses have different limitations. here you can find out more about it.

If you just have Power BI license, then at the moment of writing this article, the only way for you to use CDS, is to pay for Power Apps licenses. However, remember, for getting data from CDS, you don’t need the license (because someone already created the CDS and is paying for it). You would need a license if you are writing into CDS (through dataflows maybe, which needs its own blog article, which I’ll explain later in another post).

Summary

Common Data Service (CDS) is a database system. this database system stores the data in Azure data lake storage (cloud), and you can work with it through a management portal. CDS is a free database if you are using Power Apps licenses, You can then analyze the data of CDS using Power BI. CDS can be also used as a data warehouse layer using dataflows. In other blog articles, I’ll explain more about that scenario.

REZA RAD

TRAINER, CONSULTANT, MENTOR

Reza Rad is a Microsoft Regional Director, an Author, Trainer, Speaker and Consultant. He has a BSc in Computer engineering; he has more than 20 years’ experience in data analysis, BI, databases, programming, and development mostly on Microsoft technologies. He is a Microsoft Data Platform MVP for nine continuous years (from 2011 till now) for his dedication in Microsoft BI. Reza is an active blogger and co-founder of RADACAD. Reza is also co-founder and co-organizer of Difinity conference in New Zealand.
His articles on different aspects of technologies, especially on MS BI, can be found on his blog: https://radacad.com/blog.
He wrote some books on MS SQL BI and also is writing some others, He was also an active member on online technical forums such as MSDN and Experts-Exchange, and was a moderator of MSDN SQL Server forums, and is an MCP, MCSE, and MCITP of BI. He is the leader of the New Zealand Business Intelligence users group. He is also the author of very popular book Power BI from Rookie to Rock Star, which is free with more than 1700 pages of content and the Power BI Pro Architecture published by Apress.
He is an International Speaker in Microsoft Ignite, Microsoft Business Applications Summit, Data Insight Summit, PASS Summit, SQL Saturday and SQL user groups. And He is a Microsoft Certified Trainer.
Reza’s passion is to help you find the best data solution, he is Data enthusiast.

What is L1, L2, and L3 Support Engineering?

27 January 2022 Administrator

Harshana Madusanka Jayamaha

In this article, I’m going to explain about the Software support engineering role with my experience. I was a Level 2 and 3 support Engineer during my career.

L1 — Level 1
L2 — Level 2
L3 — Level 3
Ticket — Incident

L1 support includes interacting with customers, understand their issue and create tickets against it. The ticket then routed to the relevant L2 support ( Integration support, Server & Storage support, etc …). L1 support Engineers have basic knowledge of product/service and skill to troubleshoot a very basic issue like password reset, software installation/uninstallation/reinstallation.

L2 support manages the tickets which routed to them by L1( L2 support also can create tickets against any issue noticed by them). They have more knowledge, more experience in solving related complex issues and can guide/help L1 support folks job in troubleshooting. If the solution not provided at this level then escalate to the L3.

L3 is the last line of support and usually comprise of a development team which addresses the technical issues. They are expert in their domain and handle the most difficult problems. They do the code changes, research and develop the solution for challenging new or unknown issues.

What is Power Apps?

03 August 2021 Administrator

Power Apps is a suite of apps, services, connectors and data platform that provides a rapid application development environment to build custom apps for your business needs. Using Power Apps, you can quickly build custom business apps that connect to your business data stored either in the underlying data platform (Microsoft Dataverse) or in various online and on-premises data sources (SharePoint, Microsoft 365, Dynamics 365, SQL Server, and so on).

Power Apps.

Apps built using Power Apps provide rich business logic and workflow capabilities to transform your manual business processes to digital, automated processes. Further, apps built using Power Apps have a responsive design, and can run seamlessly in browser or on mobile devices (phone or tablet). Power Apps "democratizes" the custom business app building experience by enabling users to build feature-rich, custom business apps without writing code.

Power Apps also provides an extensible platform that lets pro developers programmatically interact with data and metadata, apply business logic, create custom connectors, and integrate with external data.

For more information:

Watch Power Apps demos
Watch videos on the Power Apps channel on YouTube

Power Apps for app makers/creators

Using Power Apps, you can create three types of apps: canvas, model-driven, and portal. More information: Overview of creating apps in Power Apps.

To create an app, you start with make.powerapps.com.

Power Apps Studio is the app designer used for building canvas apps. The app designer makes creating apps feel more like building a slide deck in Microsoft PowerPoint. More information: Generate an app from data
App designer for model-driven apps lets you define the sitemap and add components to build a model-driven app. More information: Design model-driven apps using app designer
Power Apps portals Studio is a WYSIWYG design tool to add and configure webpages, components, forms, and lists. More information: Power Apps portals Studio anatomy

Ready to convert your ideas into an app? Start here: Planning a Power Apps project

Power Apps for app users

You can run apps that you created, or that someone else created and shared with you, in browser or on mobile devices (phone or tablet). More information:

Power Apps for admins

Power Apps admins can use Power Platform admin center (admin.powerplatform.microsoft.com) to create and manage environments, get real-time, self-help recommendations and support for Power Apps and Power Automate, and view Dataverse analytics. More information: Administer Power Platform

Power Apps for developers

Developers are app makers who can write code to extend business app creation and customization. Developers can use code to create data and metadata, apply server-side logic using Azure functions, plug-ins, and workflow extensions, apply client-side logic using JavaScript, integrate with external data using virtual entities and webhooks, build custom connectors, and embed apps into your website experiences to create integrated solutions. More information:

Power Apps and Dynamics 365

Dynamics 365 apps, such as Dynamics 365 Sales, Dynamics 365 Customer Service, Dynamics 365 Marketing also use the underlying data platform (Dataverse) used by Power Apps to store and secure data. This enables you to build apps using Power Apps and Dataverse directly against your core business data already used within Dynamics 365 without the need for integration. More information: Dynamics 365 and Dataverse

Try Power Apps for free

You can build Power Apps for free. Simply sign-in to Power Apps. For more information, go to Sign in to Power Apps for the first time. Initially, you'll have access to the default environment.

A license is needed to play the apps made with Power Apps. You can both build and play Power Apps for free by signing up either for a 30 day trial or developer plan.

Purchase Power Apps

If you have decided to purchase Power Apps, see here for detailed information: Purchase Power Apps.

Power Apps US Government plans

Power Apps US Government consists of several plans for US government organizations to address the unique and evolving requirements of the United States public sector. The Power Apps GCC environment provides compliance with federal requirements for cloud services, including FedRAMP High, DoD DISA IL2, and requirements for criminal justice systems (CJI data types). More information: Power Apps US Government

What is the Power Platform?

03 August 2021 Administrator

Microsoft rapidly innovate, update and release new products and solutions, which can make staying on top of changes difficult. However, this frequent pace of innovation makes Microsoft technologies very exciting. One of the biggest new areas from Microsoft that you will read a lot about is the Power Platform.

Note: This article has been updated as Flow has been re-branded as Power Automate. Some visuals remain branded as Flow.

What is the Power Platform?

The ‘Power Platform’ is a collective term for three Microsoft products: Power BI, PowerApps and Power Automate (previously known as Flow). They provide the means to help people easily manipulate, surface, automate and analyse data and can be used with Office 365 and Dynamics 365 (as well as other third-party apps and other Microsoft services). The Power Platform is possible thanks to the Common Data Service (or CDS), which is essentially the underlying data platform that provides a unified and simplified data schema so that applications and services can inter-operate.

Power Platform Overview

Why is the Power Platform so important?

In this digital age, we are extremely reliant on data – and the amount of data companies are creating is continually increasing. While all this data is inevitable, it is useless unless companies gain insights and meaning from it - to gain tangible value.

Historically, data analysis, app creation or automation would be achieved by IT/Development teams. This would require staff to outline their requirements and aims, submit these requests to their IT department (or even an external partner) and then see whether it was approved and subsequently, wait for it to be built. This would be time-consuming and would use valuable resources internally or be costly if fulfilled externally. What're more, those requesting the solution would tend to have an immediate need and waiting for weeks could cause internal delays.

This is why the Power Platform is so exciting. The Power Platform enables data democratisation – the ability for digital information to be accessible to the typical (non-technical) end user. It provides three technologies that allow staff to do more with their data themselves without coding knowledge. While it doesn’t allow the intricacies and flexibility of custom coding, it does provide a simple method for most users to be able to create, automate or analyse their data in ways which have never been possible for the average worker.

PowerApps, Power Automate & Power BI explained

As mentioned, the Power Platform consists of three technologies - let's look at these three in more detail.

PowerApps is a low-code approach to custom app development, allowing users to quickly create apps with a ‘point and click’ approach. It allows you to:

Build mobile-friendly apps quickly and without development knowledge and reducing pressure on busy IT teams
Connect to and surface data from your business applications, such as Dynamics 365 and Office 365 (and also third-party apps)
Surface key data into a user-friendly app to help data entry – meaning users only see the information they need to fulfil a particular task

Power Automate used to be known as Microsoft Flow. This allows you to create automated workflows between your Microsoft services or other third-party applications, which allows staff to avoid carrying out repetitive tasks and save valuable time. It allows you to:

Use pre-built automation templates for common automations (within the Flow Gallery)
Create you own automations by connecting various applications, such as Outlook, SharePoint Dynamics 365 or non-Microsoft apps like Twitter, Asana, Gmail, MailChimp etc.
Set up triggers, alerts, automated emails, push notifications and much more – with no coding and in minutes
Overall it allows you to save time, reduce human error and streamline your processes

Power BI is a business analytics tool which allows you to easily connect to data sources, create visuals and gain business intelligence quickly. It allows you to:

Click and connect with Microsoft and third-party cloud services, as well as on-premise data sources
Easily manipulate data and create visuals, such as charts, dashboards, maps and many more – so you can present your data in an easy-to-digest format
Use natural language to query data and get results (i.e. “show me our sales pipeline for 2018 by month, by sales person”)
Overall, allowing you to easily analyse and make sense of complex data to enable continual improvement

How does the Power Platform fit with the wider Microsoft strategy?

The Power Platform connects to a wide range of data sources – including third-party apps such as Google Analytics and Twitter, however it is extremely powerful when working with Office 365 and Dynamics 365. Microsoft pitch the Power Platform as a way to “unlock the potential of Dynamics 365 and Office 365 faster than you ever thought possible” so you can easily extend, customise and integrate these services.

The Power Platform is going to be a big area of investment from Microsoft and as it is still fairly new, we expect that it will be regularly updated and improved as the products mature. What’s more, as Microsoft continue to focus on bringing all their technologies closer together, we can expect the Power Platform to be a key player for this – helping connect services like Microsoft 365, Dynamics 365 and Azure easily with a low-code/no-code approach.

How do I get it? Power Platform licensing and pricing

There is not currently a licensing bundle for all three products together. Instead you can purchase the three separately (and can mix and match) - or, if you haven't tried them you can use free versions or free trials to test their capabilities.

PowerApps and Power Automate licensing is fairly complex. You can read the latest licensing guidance here: https://docs.microsoft.com/en-us/power-platform/admin/powerapps-flow-licensing-faq If you have any questions around licensing and pricing we recommend speaking to a Microsoft Partner.

Power BI – There are three versions available, Free, Pro and Premium, which you can compare here: Power BI Pricing (Microsoft).

Free
Pro - note, required for sharing Power BI reports and dashboards
Premium (POA)

INSIGHT - 3 JUN 2019 - LISA CURRY