Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Your request has been blocked due to a network policy.

If you're running a script or application, please register or sign in with your developer credentials here.

Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.

if you're supplying an alternate User-Agent string, try changing back to default as that can sometimes result in a block.

if you think that we've incorrectly blocked you or you would like to discuss easier ways to get the data you want, please file a ticket here.

Extracting and structuring text elements with high accuracy using small modelsImage generated by an AI by the authorIn this post, I’ll introduce a paradigm recently developed at Anaplan for extracting temporal information from natural language text, as part of an NLQ (natural language query) project. While I will focus on time extraction, the paradigm is versatile and applicable for parsing various unstructured texts and extracting diverse patterns of information. This includes named entity recognition, text-to-SQL conversion, quantity extraction, and more.The paradigm's core lies in constructing a flexible pipeline, which provides maximal flexibility, making it easy to fine-tune a model to…

Have you encountered a problem where a 1 GB transformer-based model increases even up to 8 GB when deployed using Docker containerization?Continue reading on Towards Data Science »

Or how to “eliminate” human annotatorsImage generated by DALL·EMotivationHigh-level overview of InstructGPT with human annotated outputs and ranking for supervised learning and reward model training | Source: Training language models to follow instructions with human feedback.As Large Language Models (LLMs) revolutionize our life, the growth of instruction-tuned LLMs faces significant challenges: the critical need for vast, varied, and high-quality datasets. Traditional methods, such as employing human annotators to generate datasets — a strategy used in InstructGPT (image above)— face high costs, limited diversity, creativity, and allignment challenges. To address these limitations, the Se…

The final chapter in a comprehensive, practical guide to real-world applications of CLV analysis & predictionContinue reading on Towards Data Science »

Four advanced tricks to give your data science and machine learning classes the edge you never knew they neededContinue reading on Towards Data Science »

A step-by-step guide on building a team of AI agents that automate and refine the search and selection process matching job seeker’s skillsContinue reading on Towards Data Science »

The digital world can’t exist without the natural resources to run it. What are the costs of the tech we’re using to build and run AI?Photo by ANGELA BENITO on UnsplashThere’s a core concept in machine learning that I often tell laypeople about to help clarify the philosophy behind what I do. That concept is the idea that the world changes around every machine learning model, often because of the model, so the world the model is trying to emulate and predict is always in the past, never the present or the future. The model is, in some ways, predicting the future — that’s how we often think of it — but in many other ways, the model is actually attempting to bring us back to the past.I like t…

A beginner-friendly introduction with Python codeThis is the 3rd article in a larger series on Full Stack Data Science (FSDS). In the previous post, I introduced a 5-step project management framework for building machine learning (ML) solutions. While ML may bring to mind fancy algorithms and technologies, the quality of an ML solution is determined by the quality of the available data. This raises the need for data engineering (DE) skills in FSDS. This article will discuss the most critical DE skills in this context and walk through a real-world example.Photo by the blowup on UnsplashFull Stack Data Science (FSDS) involves managing and implementing ML solutions end-to-end. Data engineering…

Top 3 lessons learned: the problem, the size, and the dataPicture by Snapwire, on PexelsThis blog post is an updated version of part of a conference talk I gave on GOTO Amsterdam last year. The talk is also available to watch online.As a Machine Learning Product Manager, I am fascinated by the intersection of Machine Learning and Product Management, particularly when it comes to creating solutions that provide value and positive impact on the product, company, and users. However, managing to provide this value and positive impact is not an easy job. One of the main reasons for this complexity is the fact that, in Machine Learning initiatives developed for digital products, two sources of un…

8 years ago I started my bachelor’s degree in Geography. Now I’m a Data Scientist; this is the story of how (and why) I’ve got hereContinue reading on Towards Data Science »

Looked at from another angle, there is much more noise than signal in financial data.

Another financial market application of LLMs might be synthetic data creation [4,8].

Then precious real market data could be employed to fine-tune the predictions and determine precisely the optimal speed to trade.

Financial market practitioners are often interested in extreme events, the times when trading strategies are more likely to experience significant gains or losses.

CitationFor attribution in academic contexts or books, please cite this work asRichard Dewey and Ciamac Moallemi, "Financial Market Applications of LLMs," The Gradient, 2024

All of these terms (“AI”, “gender”, and “bias”) can be somewhat overused and ambiguous.

A Short History of Studying Gender Bias in AIHere, I cover a very small sample of papers I’ve found influential studying gender bias in AI.

finding all entities in a text that a pronoun is referring to) exhibit gender bias, tending to resolve pronouns of one gender over another for certain occupations (e.g.

This article mainly focused on gender bias — and particularly, on binary gender.

AcknowledgementsThis post was originally posted on Art Fish IntelligenceCitationFor attribution in academic contexts or books, please cite this work asYennie Jun, "Gender Bias in AI," The Gradient, 2024@article{Jun2024bia…

As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.

Here we’ll discuss:The advantages (and disadvantages) of Mamba (🐍) vs Transformers (🤖),Analogies and intuitions for thinking about Mamba, andWhat Mamba means for Interpretability, AI Safety and Applications.

The Mamba BlockLike a Transformer made up of stacked transformer blocks, Mamba is made up of stacked Mamba blocks as above.

The Mamba authors write, “the efficiency vs. effectiveness tradeoff of sequence models is characterised by how well they compress their state”.

Thanks to Gonçalo for reading an early draft, Jaden for the nnsight library …

We've just seen 3 prominent families of LLM usage in self-driving cars: Perception, Planning, and Generation.

The first wave of papers mentioning LLMs in Self-Driving Cars is from mid-2023, so let's give it some time.

Next StepsIf you want to get started on LLMs for self-driving cars, there are several things you can do:⚠️ Before this, the most important : If you want to keep learning about self-driving cars.

Author BioJérémy Cohen is a self-driving car engineer and founder of Think Autonomous, a platform to help engineers learn about cutting-edge technologies such as self-driving cars and advanced Computer Vision.

CitationFor attribution in academic contexts or books, please cite this work…

Beyond the requirements of semantic similarity, there are no constraints on what embedding must be assigned for a given text input.

What if someone hacks into the database and gains access to all your text embedding vectors – would this be bad?

From text to embeddings...back to textThe problem of recovering text from embeddings is exactly the scenario we tackle in our paper Text Embeddings Reveal As Much as Text (EMNLP 2023).

Scaling and future workThe fact that text embeddings can be perfectly inverted raises many follow-up questions.

CitationFor attribution in academic contexts or books, please cite this work asJack Morris, "Do text embeddings perfectly encode text?

If your model latches on to these during training, it will appear to work well, but may not work on new data.

This happens when the model training pipeline has access to information it shouldn’t have access to, particularly information that confers an advantage to the model.

This means that knowledge of the test data is implicitly entering the model training pipeline, even if it is not explicitly used to train the model.

Well, this is a common thing to do, but if you’re developing a model iteratively and using the same test set to evaluate the model after each iteration, then you’re basically using that test set to guide the development of the model.

CitationFor attribution in academic cont…

Evolution of single-cell sequencing over timeHaving explored the panorama of single-cell sequencing, let us now delve into the role of deep learning in the context of single-cell sequencing.

Deep Learning on single-cell sequencingDeep learning is increasingly employed in single-cell analysis due to its capacity to handle the complexity of single-cell sequencing data.

The deep learning approach, however, autonomously captures relevant characteristics from single-cell sequencing data, addressing the heterogeneity between single-cell sequencing experiments, as well as the associated noise and sparsity in such data.

As we explore the reasons behind using deep learning in single-cell sequencing …

In order to obtain a license or permit from FERC, hydroelectric dam operators must submit detailed plans and studies demonstrating that their facility meets regulations.

Typically, a hydroelectric dam requires lots of space to store water on one side of it, which means they tend to be located away from population centers.

Enter Computer VisionSome organizations are exploring the use of computer vision and machine learning to significantly automate fish counting.

The annotated images are then used to train a machine learning model.

CitationFor attribution of this in academic contexts or books, please cite this work as:Kevin McCraney, "Salmon in the Loop", The Gradient, 2023.

In recent work with computer networking and machine learning collaborators from ETH Zürich, we studied the applicability of neural algorithmic reasoning in computer networking [27].

Our proposal, the neural algorithmic reasoning blueprint [32], aims to bridge this divide by neuralising the target algorithm.

Neural Algorithmic Reasoning with Causal Regularisation.

Neural Algorithmic Reasoning.

CitationFor attribution in academic contexts or books, please cite this work asPetar Veličković, "Neural Algorithmic Reasoning", The Gradient, 2023.

This community has developed an extensive vocabulary around theories of AI safety and alignment, many first introduced as detailed blog posts in forums like LessWrong and AI Alignment Forum.

One such idea that is useful for contextualizing technical alignment work — and is perhaps the more formal version of what Bostrom was referring to — is the concept of intent alignment.

Anthropic’s product marketing pages are plastered with notes and phrases about their alignment work —“HHH” is also Claude's biggest selling point.

The site uses the phrasing “AI Safety” instead of “AI Alignment” in the title, but the article itself proceeds to use “safety” and “alignment” interchangeably without differen…

This brief introduction is aimed at those working within the AI community who are interested in AI consciousness, but may not know much about the philosophical and scientific work behind consciousness generally or the topic of AI consciousness in particular.

(Image by author)AI and ConsciousnessTwo Problems for AI ConsciousnessLet’s return to the topic of AI consciousness.

The problem of AI consciousness may seem less difficult than the hard problem: the problem of AI consciousness only asks if silicon could support consciousness, but it does not ask for an explanation of why silicon can or cannot, like the hard problem does.

The second test proposed by Schneider and Edwin Turner [27], call…

We’ve identified three key areas where such programs can level up: dataset curation, a pattern language for usability, and filtering.

The format for a friction hinge pattern might look like this:Pattern Name Friction Hinge Pattern Description The Friction Hinge pattern addresses the need for adjustable friction in hinges so as to provide tuneable resistance, but without compromising smooth movement.

Alas, once text-to-CAD models get open-sourced or leaked, many of these queries will be satisfied without compunction.

____In conclusion, the emergence of AI-powered text-to-CAD generation presents both risks and opportunities, the ratio of which is still very much undecided.

CitationFor attribu…

In this piece, I will discuss the tendency towards “interpretability creationism” – interpretability methods that only look at the final state of the model and ignore its evolution over the course of training – and propose a focus on the training process to supplement interpretability research.

In the case of language models, they behave similarly to ngram models early on and exhibit linguistic patterns later.

Let us consider an explanation usually based on analyzing static models: hierarchical behavior in language models.

An ExampleI recently had to manage the trap of interpretability creationism myself.

Citation:For attribution in academic contexts or books, please cite this work asNaomi …

Understanding what LLMs learn about linguistics from large-scale pretraining is a good framework for understanding how they work in general.

How we extend in-context learning to structured prediction tasks with structured prompting (left) and an example of using structured prompting with an LLM to annotate parts-of-speech (right).

Overall, structured prompting can consistently generate the linguistic structure underlying text from LLMs, confirming that these models have implicitly learned these structures during pretraining.

This leads us to ask: what factors of structured prompting allow LLMs to annotate linguistic structure?

By searching for labels instead of full examples, we can find ta…

Created Using IdeogramPretraining Large Language Models (LLMs) on massive text datasets has become the norm.

When these LLMs are applied to specific tasks, it’s often necessary to integrate additional information, such as the latest news or specialized knowledge, into the already trained model.

The main strategies considered are in-context learning through RAG and supervised fine-tuning.

On the other hand, supervised fine-tuning aims to identify broader patterns in the documents, which could lead to better performance in tasks and alignment with user needs.

However, this method may not always take advantage of documents during the testing phase or may overlook errors in document retrieval.

Created Using DALL-EIn this Issue:An overview of function calling in LLMs and its role in autonomous agnets.

An introduction to the Phidata framework for building autonomous agents.

💡 ML Concept of the Day: LLM Function Calling and Autonomous AgentsIn the first few installments of this series about autonomous agents, we have been exploring the ability of agents to integrate with third party tools or APIs.

Function calling refers to the ability of LLMs to invoke functions from external APIs.

In the context of autonomous agents, function calling plays a role by allowing agents to retrieve information or perform actions on external systems.

Again, not that small, but small enough ;) Apple open-sourced OpenELM, a family of LLMs optimized for mobile scenarios.

After all, nobody likes a know-it-all ;)"🔎 ML ResearchPhi-3Microsoft Research published the technical report of Phi-3, their famous small language model that excel at match and computer science task.

The method is an interesting approach to interpretability to prove generative AI models to undestand their behavior —> Read more.

LayerSkipMeta AI Research published a paper introducing LayerSkip, a method for accelerated inference in LLMs.

🤖 Cool AI Tech ReleasesOpenELMApple open sourced OpenELM, a family of small LLMs optimized to run on devices —> Read more.

Created Using IdeogramThe open-source generative AI landscape is experiencing tremendous momentum.

Innovation comes not only from startups like HuggingFace, Mistral, or AI21 but also from large AI labs such as Meta.

Databricks has been one of the tech incumbents exploring different angles in open source generative AI, mainly after the acquisition of MosaicML.

A few days ago, Databricks open sourced DBRX, a massive general-purpose LLM that show incredible performance across different benchmarks.

Databricks released both the baseline model DBRX Base as well as the intstruction fine-tuned one DBRX Instruct.

Created Using IdeogramIn this Issue:An overview of large action models(LAM) in autonomous agents.

An introduction to the MetaGPT framework for building autonomous agents.

💡 ML Concept of the Day: Large Action Models and Autonomous AgentsThe ability to execute actions in a given environment is one of the hallmarks of autonomous agents.

One of the key topics of debate in autonomous agent circles is how much of the action execution relies on external components versus being built into the model itself.

One of the most interesting approaches among the proponents of the latter camp is what is known as large action models (LAMs).

You can subscribed to The Sequence below:📝 Editorial: Some Cool Details About Llama 3I had an editorial prepared for this week’s newsletter, but then Meta AI released Llama 3!

I prefer to use the term "open models," given that these releases are not completely open source, but that’s just my preference.

The release of Llama 3 builds on incredible momentum within the open model ecosystem and brings its own innovations.

The momentum in the generative AI open models space definitely continues, even if it forced me to rewrite the entire editorial.

🤖 Cool AI Tech ReleasesLlama 3Meta AI introduced the highly anticipated Llama 3 model —> Read more.

Created Using IdeogramVideo games have long served as some of the best environments for training AI agents.

However, most of the AI breakthroughs in 3D game environments have been constrained to one or a small number of games.

The goal of the project was to develop instructable agents that can interact with any 3D environment just like a human by following simple language instructions.

Language is the most powerful and yet simple abstraction for communicating instructions about the world or, in this case, a 3D virtual world.

The magic of SIMA is its ability to translate those abstract instructions into mouse and keyboard actions used to navigate an environment.

Created Using IdeogramIn this Issue:Tool learning in autonomous agents.

💡 ML Concept of the Day: Tool Learning in Autonomous AgentsOne of the key differentiators between agents and models is the capability of the former to take actions in a given environment.

Part of that action execution typically involves interactions with different systems or tools.

From this perspective, tool learning has become one of the most important building blocks of autonomous agents.

When it comes to tool learning in autonomous agents, we should identify two main groups of interactions:

You can subscribed to The Sequence below:📝 Editorial: Neuro-Symbolic Models are Making a ComebackLarge language models (LLMs) have dominated the AI narrative in recent years to the point that we almost need to wonder about the future of other areas of machine learning.

One of the most interesting options to address these limitations comes from a pretty old ML school: neuro-symbolic models.

As the name indicates, neuro-symbolic models combine neural networks, such as LLMs, with smaller, easier-to-interpret symbolic models to adapt LLMs to specific domains.

Neuro-symbolic architectures are making a comeback as one of the possible architectures to play a role in this movement.

Neuro-symbolic m…

Created Using DALL-EThe Chinese ecosystem around foundation models have been on fire recently.

One of the most ambitious foundation models effort in China comes from 01, the startup founded by former Microsoft and Google researcher Kai Fu Lee.

01’s first iteration came in the form of the Yi models.

A few days ago, 01 published a technical report about the Yi models and we thought it would be interesting to share some details.

The Yi series models stands out for their bilingual capabilities.

Created Using IdeogramIn this Issue:Building LLM-based vs. computer-based autonomous agents.

💡 ML Concept of the Day: The Two Big Schools for Building Autonomous AgentsIn our series about autonomous agents, today we would like to explore the two fundamental schools used for implementations of this AI systems.

Typically, we associate autonomous agents with LLMs but there are competitive techniques fundamentally based on computer vision models which has been gaining quite a bit of traction.

CV-based autonomous agents fundamentally focus on recording actions in a user’s computer and replicating those actions with models that can understand pixel-by-pixel positions and mouse actions.

In general…

You can subscribed to The Sequence below:📝 Editorial: Generative Audio Models Just Had a Great WeekAudio is rapidly becoming one of the most important frontiers in generative AI, a field that is advancing swiftly.

Technically, generative audio poses a fundamentally simpler problem than video or 3D, which leads to faster iterations in research and implementation.

The pace of innovation in generative audio is accelerating at remarkable levels.

To these innovations, you need to add more established players such as Eleven Labs, which have been pushing the boundaries of generative audio for years.

OpenAI Custom ModelsOpenAI announced enhacements to its training API as well as new mechanisms for …

This year the EU AI Act came into law and has worried a lot of developers.

Anyone who is selling an AI product within the EU has to comply with the EU AI Act, even if you’re not based in the EU.

Minimal Risk AI Applications: This category covers the majority of AI applications such as AI in games, spam filters and recommendation engines.

The EU AI Act creates a separate category for what they consider to be “General Purpose AI” systems.

The worst elements of early drafts have mostly been stripped from the EU AI Act.

Created Using IdeogramThe pace of research in generative AI is nothing short of remarkable.

Even so, from time to time, there are papers that literally challenge our imagination about how far the generative AI space can go.

A few weelks ago, Google DeepMind published some of that work with the release of Genie, a model that is able to generative interactive game environments from text and images.

This is the vision that Google DeepMind has turned into reality with Genie, a groundbreaking approach to generative AI.

Genie can craft interactive environments from a mere text or image prompt, thanks to its training on over 200,000 hours of publicly available gaming videos from the Internet.

Created Using IdeogramIn this Issue:An review of the key capabilities of autonomous agents.

An introduction to the Crew AI framework for building autonomous agents.

Different frameworks have outlined various feature sets for autonomous agents.

However, there are a few building blocks that can be consistently considered as the strong foundation of any autonomous agent application.

Most capabilities of autonomous agents can be considered variations of the following key areas:

In recent advancements, the fusion of video foundation models and large language models has emerged as a promising avenue for constructing robust video understanding systems, transcending the constraints of predefined vision tasks. However, while these methods exhibit commendable performance on shorter videos, they encounter significant hurdles when confronted with longer video sequences. The escalating computational complexity and memory demands inherent in sustaining long-term temporal connections pose formidable challenges.In a new paper MovieChat+: Question-aware Sparse Memory for Long Video Question Answering, a pioneering research group introduces MovieChat, a novel framework tailored…

Large language models (LLMs) endowed with long-context capabilities, such as GPT-4 and Gemini, are increasingly finding versatile applications in various domains like chatbots, vision generation, and financial analysis. However, their efficacy is hampered by the inefficient utilization of computational resources and a substantial memory footprint, particularly when tasked with generating long sequences.Addressing these challenges, in a new paper TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding, a research team from Carnegie Mellon University and Meta AI introduces TriForce — a hierarchical speculative decoding system tailored for scalable lo…

In recent years, there has been a surge in the development of large language models (LLMs) tailored for code-related tasks. These LLMs have shown remarkable proficiency in aiding developers with tasks such as writing, editing, explaining, and reviewing code. However, they often stumble when faced with more intricate software engineering challenges that demand a deeper understanding of a program’s runtime behavior.Addressing this gap, in a new paper NExT: Teaching Large Language Models to Reason about Code Execution, a Google DeepMind research team proposes Naturalized Execution Tuning (NExT), a method aims to equip LLMs with the ability to scrutinize program execution traces and deduce runt…

AlphaFold2 (AF2), crafted by DeepMind, stands as a beacon in the realm of artificial intelligence (AI), boasting the remarkable ability to predict the three-dimensional (3D) structures of proteins from amino acid sequences with unprecedented atomic-level precision. While lauded as a revolutionary advancement in protein folding, its training regimen has long been hampered by its laborious nature, failing to reap significant benefits from the scaling up of computational resources.In a new paper ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours, a team of researchers from NVIDIA presents ScaleFold, a novel and scalable training methodology tailored for the AlphaFold model. Notabl…

In the expansive realm of artificial intelligence and natural language processing, Small Language Models (SLMs) are making significant strides. Unlike their larger counterparts with hefty parameter counts and demanding computational needs, SLMs are sleeker versions crafted for optimal performance even in resource-constrained settings.In a new paper RecurrentGemma: Moving Past Transformers for Efficient Open Language Models, a Google DeepMind research team introduce RecurrentGemma, an open language model built on Google’s innovative Griffin architecture. This model reduces memory usage and facilitates efficient inference on lengthy sequences, thereby unlocking new possibilities for highly ef…

Efficient on-device neural networks offer rapid, real-time, and interactive experiences while safeguarding private data from public internet exposure. Yet, the computational limitations of mobile devices present a formidable challenge in maintaining a delicate balance between accuracy and efficiency.Addressing this challenge head-on, a recent paper titled “MobileNetV4 — Universal Models for the Mobile Ecosystem,” penned by a Google research team, unveils the latest iteration of MobileNets: MobileNetV4 (MNv4). This cutting-edge model boasts an impressive 87% ImageNet-1K accuracy, coupled with an astonishingly low Pixel 8 EdgeTPU runtime of merely 3.8ms.At the heart of this breakthrough lies …

Transformer-based language models have emerged as powerful tools across various tasks, underlining their significance in critical contexts. Understanding the inner workings of these models is paramount for ensuring their safety, reliability, and trustworthiness, given their widespread adoption.In a new paper LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models, a research team from Meta, University College London and Universitat Politècnica de Catalunya introduces the LM Transparency Tool (LM-TT), an open-source interactive toolkit designed for dissecting Transformer-based language models.Existing analysis tools often focus on isolated aspects of decision-making …

The Large Language Model (LLM) has showcased remarkable efficacy across various real-world applications, including intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Nonetheless, the current methodologies for on-device deployment of LLMs are hampered by sluggish inference speeds, leading to subpar user experiences.In a new paper Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs, researchers from OPPO AI Center have introduced a solution. They present four optimization techniques and introduce a novel mobile inference engine dubbed Transformer-Lite. This engine outperforms CPU-based FastLLM and GPU-bas…

Revolutionizing Video Understanding: Real-Time Captioning for Any Length with Google’s Streaming ModelThe exponential growth of online video platforms has led to a surge in video content, thereby heightening the need for advanced video comprehension. However, existing computer vision models tailored for video understanding often fall short, typically analyzing only a limited number of frames, typically spanning mere seconds, and categorizing these brief segments into predefined concepts.To address the abovementioned challenge, in a new paper Streaming Dense Video Captioning, a Google research team proposes a streaming dense video captioning model, which revolutionizes dense video captioning…

AURORA-M: A Global Symphony of Innovation as 33 Prestigious Institutions Unify for Open-Source Multilingual MasteryLarge Language Models (LLMs) have revolutionized various applications, including machine translation, text summarization, dialogue systems, and code generation. Yet, the hefty computational requirements for pretraining these models pose significant barriers to broader accessibility and development.To address these challenges, recent open-source initiatives like BLOOM, StarCoder, and StarCoder-2 have emerged, aiming to democratize access to pretrained LLMs. However, these models encounter limitations such as restricted multilingual capabilities, computational intensity, and the …

Huawei & Peking U’s DiJiang: A Transformer Achieving LLaMA2–7B Performance at 1/50th the Training CostThe Transformer architecture has emerged as a pivotal tool in numerous domains, excelling particularly in tasks like speech recognition, machine translation, and document summarization. Yet, its efficacy often hinges on expanding the model’s size to tackle increasingly intricate challenges, thereby imposing substantial computational burdens.In the pursuit of alleviating the computational strain associated with Transformers, the exploration of linear attention mechanisms has garnered notable traction. Nonetheless, enhancing these mechanisms typically entails extensive retraining, a prohibiti…

The transformer architecture has emerged as the predominant framework for deep learning, playing a pivotal role in the remarkable achievements of large language models like ChatGPT. Despite its widespread adoption, the theoretical underpinnings of its success remain largely uncharted territory.In a new paper The Topos of Transformer Networks, a King’s College London research team delves into a theoretical exploration of the transformer architecture, employing the lens of topos theory. This innovative approach conjectures that the factorization through “choose” and “eval” morphisms can yield effective neural network architecture designs.The primary objective of this paper is to offer a categ…

In recent years, there has been a remarkable surge in the effectiveness of large transformer models trained through generative modeling on extensive language datasets sourced from the Internet. These models have demonstrated impressive capabilities across diverse domains. By predicting subsequent words, these models glean intricate understandings of language, which can then be applied to various tasks through multi-task learning and efficient few-shot learning techniques.This success has led researchers to ponder: can we replicate this approach to develop robust models for sensory and motor representation? While there have been encouraging signs of progress in learning sensorimotor represen…

Despite the remarkable capabilities of contemporary large language models like GPT-4, Claude 2, or Gemini, the inner mechanisms governing their operations remain shrouded in mystery. While the intricate details of these models are kept under wraps, they are made accessible through APIs, prompting the question: How much insight can adversaries glean about these models by interacting with their APIs?To answer this question, in a new paper Stealing Part of a Production Language Model, a research team from Google DeepMind, ETH Zurich, University of Washington, OpenAI and McGill University introduces a groundbreaking model-stealing attack. This attack unveils precise, nontrivial information from…

In recent years, there has been a remarkable surge in the development of generative AI, showcasing the potential of models to create novel and imaginative content. While strides have been made in various domains, the realm of video generation stands as a promising frontier.Recent advancements suggest that scaling up these models could further enhance their capabilities. However, there remains a significant gap between the interactive engagement offered by video generative models and the rich interactions facilitated by language tools like ChatGPT, not to mention more immersive experiences.In response to this challenge, in a new paper Genie: Generative Interactive Environments, a research te…

Или кликбейт — и это в Nature?

Статья ниже — подробный разбор достаточно сложного топика, и в некоторых моментах нужно будет сосредоточиться и вдумчиво читать.

Они же пишут код в помощь разработчикам, да и в целом помогают решать разного рода проблемы.

Однако может так получиться, что сет долгое время не выпадает — и не потому, что игроки проворонили, а потому, что его действительно просто нет на столе.

Надеемся, что после прочтения этой статьи стало ясно, что ошибки нейросетей — это не баг, это фича.

Также присоединяйтесь к моему телеграм каналу AI[ex]Time , где я пишу про машинное обучение, NLP, LLM и в том числе про агентов.

LLaVaВ качестве LLM для генерации текстового ответа используется LLaMA, которая декодирует эмбеддинги (то же, что и векторы) картинок и входного текста в ответ.

На вход модель получает картинку и запрос от пользователя (Normal prompt) и на выходе должна дать ответ (Response).

Модель таким образом от решения задачи напрямую переходит к рассуждению по шагам, что в некотором смысле и является декомпозицией задачи.

И это улучшает качество работы модели — в том числе и ризонинга.

Мы помним, что в начале 2023-го продуктом уже пользовалось больше 100 млн человек в месяц.

У самых любознательных читателей может возникнуть вопрос: а как это вообще работает?

Насколько нам известно, это первый раз, когда модель такого масштаба обучается на синтетических данных, а не на произведенных человеком.

Использование такой модели дешевле в 3 раза на текст из промпта, и в 2 раза на генерируемые токены (их обычно меньше).

Но вот как с ними обращаться, когда использовать и как комбинировать — это уже решает AI по контексту диалога.

Меня зовут Валентин Малых, я — руководитель направления NLP-исследований в MTS AI, вот уже 6 лет я читаю курс по NLP.

Поскольку я все время отвечаю одно и то же, появилась идея сделать пост про мой список книг, заодно описав их.

При этом в книге больше информации про информационный поиск (information retrieval) и меньше про NLP, но в наше время эти две области уже (или все еще) очень близки.

Правда, его тоже уже не достать, хотя PDF версия ищется без проблем.

Foundations of Statistical Natural Language ProcessingНасколько мне известно, эта книга не переводилась на русский язык.

Меня зовут Валентин Малых, я — руководитель направления NLP-исследований в MTS AI, вот уже 6 лет я читаю курс по NLP.

Поскольку я все время отвечаю одно и то же, появилась идея сделать пост про мой список книг, заодно описав их.

При этом в книге больше информации про информационный поиск (information retrieval) и меньше про NLP, но в наше время эти две области уже (или все еще) очень близки.

Правда, его тоже уже не достать, хотя PDF версия ищется без проблем.

Foundations of Statistical Natural Language ProcessingНасколько мне известно, эта книга не переводилась на русский язык.

Платформа для экспериментов в RecSysНа примере Киона я показала сложности экспериментов с рекомендательными системами.

В работе над RecSys в реальном сервисе мы строим гипотезы, выкатываем модели в АБ тесты, строим новые гипотезы.

Нашей целью была инфраструктура и пайплайны для максимально быстрых и удобных экспериментов с любыми алгоритмами.

Трекинг экспериментов: метрики и визуальный анализТрекинг экспериментов мы проводили одновременно в Mlflow и в отчётах в репозитории.

Схема на картинке:Инфраструктура платформы для экспериментов и приложения с рекомендациямиЖёлтые стрелки отражают запись результатов экспериментов: логирование метрик с кросс-валидации, сохранение обученных моделей в Min…

В первой части статьи я описывала, как мы с напарником решили выкатить модель из соревнования в онлайн рекомендации, и что из этого вышло.

Давайте построим фичу для бустинга, которая будет решать главную проблему нашей текущей модели - не релевантные айтемы в хороших подборках.

Также у нас были замечательные фичи, проверенные ещё на модели в соревновании.

В задаче классификации мы подаём модели кандидатов, бывших в интеракциях в качестве позитивных таргетов, и не бывших в качестве негативных, и учим его отличать их друг от друга.

Самый стабильный подход к кросс-валидации в рекомендательных системах - это схема скользящего окна, которая больше всего соответствует работе модели в реальном мир…

Смотрим, что мы имеем в Кионе:Распределение популярности топ 100 айтемов в датасете KionЧто там в топе просмотров?

Популярные айтемы заполняют собой полку рекомендаций и на этот запрос, и на многие другие.

Фильм занимает 6 место в топе просмотров в датасете и очень не вовремя попадает в рекомендации у самых разных алгоритмов.

Проверим рекомендации от модели с максимумом Recall без популярного:Рекомендации модели bm25 с максимумом Recall без популярного.

Recall и MAP понизились примерно в 2 раза от модели в соревновании.

Почти сразу после публикации поста про систему поиска новостей о трудовых конфликтах в СНГ я познакомился с коллективом проекта «Диалектик».

Коллектив волнуют процессы, происходящие как в экономическом базисе, так и в его надстройке – в обществе, культуре, политике.

Этот метод познания помогал всем классикам марксизма анализировать сложные ситуации в экономике и обществе, этим он ценен и для коллектива «Диалектика».

Поэтому внутри коллектива ведется разработка ИТ-инструментов для внутреннего (только для редакторов) и для внешнего (для всех пользователей) использования.

Этот публичный агрегатор находится на этапе тестирования и в скором времени будет представлен общественности.

Inpainting and outpainting have long been popular and well-studied image processing domains. Traditional approaches to these problems often relied on complex algorithms and deep learning techniques yet still gave inconsistent outputs. However, recent advancements in the form of Stable diffusion have reshaped these domains. Stable diffusion now offers enhanced efficacy in inpainting and outpainting while […]

The post Inpainting and Outpainting with Stable Diffusion appeared first on MachineLearningMastery.com.

Stable Diffusion’s latest models are very good at generating hyper-realistic images, but they can struggle with accurately generating human faces. We can experiment with prompts, but to get seamless, photorealistic results for faces, we may need to try new methodologies and models. In this post, we will explore various techniques and models for generating highly […]

The post Generate Realistic Faces in Stable Diffusion appeared first on MachineLearningMastery.com.

The deep learning model of Stable Diffusion is huge. The weight file is multiple GB large. Retraining the model means to update a lot of weights and that is a lot of work. Sometimes we must modify the Stable Diffusion model, for example, to define a new interpretation of prompts or make the model to […]

The post Using LoRA in Stable Diffusion appeared first on MachineLearningMastery.com.

Generating pictures using Stable Diffusion in all cases would involve to submit a prompt to the pipeline. This is only one of the parameters, but the most important one. An incomplete or poorly constructed prompt would make the resulting image not as you would expect. In this post, you will learn some key techniques to […]

The post Prompting Techniques for Stable Diffusion appeared first on MachineLearningMastery.com.

Launching the Stable Diffusion Web UI can be done in one command. After that, you can control the image generation pipeline from a browser. The pipeline has a lot of moving parts and all are important in one way or another. To effectively command Stable Diffusion to generate images, you should recognize the widgets from […]

The post How to Create Images Using Stable Diffusion Web UI appeared first on MachineLearningMastery.com.

The introduction of GPT-3, particularly its chatbot form, i.e. the ChatGPT, has proven to be a monumental moment in the AI landscape, marking the onset of the generative AI (GenAI) revolution. Although prior models existed in the image generation space, it’s the GenAI wave that caught everyone’s attention. Stable Diffusion is a member of the […]

The post A Technical Introduction to Stable Diffusion appeared first on MachineLearningMastery.com.

The advance of generative machine learning models makes computers capable of creative work. In the scope of drawing pictures, there are a few notable models that allow you to convert a textual description into an array of pixels. The most powerful models today are part of the family of diffusion models. In this post, you […]

The post Brief Introduction to Diffusion Models for Image Generation appeared first on MachineLearningMastery.com.

The path to uncovering meaningful insights often starts with a single step: looking at the data before asking questions. This journey through the Ames Housing dataset is more than an exploration; it’s a narrative about the hidden stories within numbers, waiting to be told. Through a “Data First Approach,” we invite you to dive deep […]

The post Unfolding Data Stories: From First Glance to In-Depth Analysis appeared first on MachineLearningMastery.com.

Data Science embodies a delicate balance between the art of visual storytelling, the precision of statistical analysis, and the foundational bedrock of data preparation, transformation, and analysis. The intersection of these domains is where true data alchemy happens – transforming and interpreting data to tell compelling stories that drive decision-making and knowledge discovery. Just as […]

The post The Da Vinci Code of Data: Mastering The Data Science Mind Map appeared first on MachineLearningMastery.com.

The real estate industry is a vast network of stakeholders including agents, homeowners, investors, developers, municipal planners, and tech innovators, each bringing unique perspectives and objectives to the table. Within this intricate ecosystem, data emerges as the critical element that binds these diverse interests together, facilitating collaboration and innovation. PropTech, or Property Technology, illustrates this […]

The post Finding Value with Data: The Cohesive Force Behind Luxury Real Estate Decisions appeared first on MachineLearningMastery.com.

Data transformations enable data scientists to refine, normalize, and standardize raw data into a format ripe for analysis. These transformations are not merely procedural steps; they are essential in mitigating biases, handling skewed distributions, and enhancing the robustness of statistical models. This post will primarily focus on how to address skewed data. By focusing on […]

The post Skewness Be Gone: Transformative Tricks for Data Scientists appeared first on MachineLearningMastery.com.

Sponsored Content In my decade of teaching online, the most significant inspiration has been that online learning democratizes access to education globally. Regardless of your ethnic background, income level, and geographical location—as long as you can surf the web—you can find an ocean of free educational content to help you learn new skills. […]

The post Best Free Resources to Learn Data Analysis and Data Science appeared first on MachineLearningMastery.com.

In the world of data science, where raw information swirls in a cacophony of numbers and variables, lies the art of harmonizing data. Like a maestro conducting a symphony, the skilled data scientist orchestrates the disparate elements of datasets, weaving them together into a harmonious composition of insights. Welcome to a journey where data transcends […]

The post Harmonizing Data: A Symphony of Segmenting, Concatenating, Pivoting, and Merging appeared first on MachineLearningMastery.com.

In the realm of data analysis, SQL stands as a mighty tool, renowned for its robust capabilities in managing and querying databases. However, Python’s pandas library brings SQL-like functionalities to the fingertips of analysts and data scientists, enabling sophisticated data manipulation and analysis without the need for a traditional SQL database. This exploration delves into […]

The post Beyond SQL: Transforming Real Estate Data into Actionable Insights with Pandas appeared first on MachineLearningMastery.com.

This means we can reduce all the crew battle outcomes down to a single \(n \times n\) matrix, which I’ll call the crew battle matrix.

So I Wrote Some Python Code to Compute \(f\) for Arbitrary Crew BattlesLet’s consider the RPS crew battle again.

Here’s the matchup matrix:and here’s what my code outputs for the crew battle matrix, assuming optimal play.

But also, this suggests that crew battle strategy really isn’t that important in the first place!

If your crew loses, you don’t get to blame bad crew battle strategy.

This has spoilers for MIT Mystery Hunt 2024.

I hunted with teammate again this year, because there is nothing quite like writing a Mystery Hunt to forge friends through fire.

I needed to spend some to avoid hitting the vacation cap, and what better time than Mystery Hunt?

If you were forced to pick how a Mystery Hunt runs long, I think most people would pick the “too many puzzles” side of Mystery Hunt 2024 over the “too difficult puzzles” side of Mystery Hunt 2023.

So did Spoilr, the codebase for Mystery Hunt 2022, and tph-site from Mystery Hunt 2023.

In August 2020, I wrote a post about my AI timelines.

Diffusion based image augmentation has been shown to improve robot learning, and Anthropic has based a lot of its branding on constitutional AI and “RL from AI feedback”.

I don’t like AI Twitter for reasons I’ve explained here, but I especially do not AI twitter post-ChatGPT.

In AI, models can never do everything people claim they can, but what the models can do is ever-growing and never slides backward.

The bull is to say that we can figure out how to scale models, and scaled up models will solve all the other hard problems.

You know how much time I have spent studying how to squeeze Neopoints water out of the Neopets stone?

I’d say you can expect to get about 25k NP a day, depending on how many NP rewards you get.

Trudy’s Surprise also gives items for 7 day streaks, but these items are usually junk and not worth anything.

When you win against CPU opponents, you earn a small amount of Neopoints and an item drop.

Or your needs to…see someone do a lot of research into things that don’t matter?

Everfree Northwest is a My Little Pony convention in the Seattle area.

Except, I heard that with the ending of BronyCon, Everfree Northwest is the largest pony convention left.

My Little Pony: Friendship is Magic is Generation 4, or G4, and is the one that kicked off the brony fandom.

People tend to assume that “pony concert” means “remixes of pony songs”, or at least “overt references to My Little Pony”.

The first was a signed copy of Ponies: The Galloping, the Magic: The Gathering x My Little Pony crossover.

I started working on that post right after Mystery Hunt 2023 finished, and hardcore focused on getting it out ASAP.

The end result was that I was exerting “write Mystery Hunt” levels of effort (10-20 hrs/week) for 1.5 years straight.

markdown 1929 2023 - 04 - 21 - mh - 2023. markdown 114 2023 - 05 - 09 - bootes - 2023. markdown 153 2023 - 07 - 19 - ml - hurry .

markdownPosts in LimboPost about Dominion Online:Odds of writing this year: 5%Odds of writing eventually: 25%My priorities are moving away from Dominion.

Post about AI timelines:Odds of writing this year: 90%Odds of writing eventually: 99%I’m not planning a big update to the last timelines post.

As I asked questions about compute resources and the presenter’s position on generative AI ethics, I had a moment of realization.

Why are we talking about whether an RTX 3090 is big enough to finetune a LLaMa checkpoint?

The much easier and less-speculative way for a field to move faster is by having more people working in that field.

The MLP AI enthusiasts mentioned some pretrained LLMs that I had not even heard of.

I remember being a young whippersnapper, in the deep learning wave of 2015.

Generative AI is a membership based concept, so it is defined by the set of approaches and applications that get put under that name.

I think this is one part of the intrigue of generative AI for science.

So maybe generative AI applications are part of this drive for a post-theory mode of science.

This generative science, it is hoped, can add vigour into the sciences, especially in cross-disciplinary ways and provide broad-based benefit.

Generative AI for Science presents opportunities for new ways of doing theory and renewed vigour across our sciences.

Keynote at the Stanford Human-centered AI 2023 Fall Conference on New Horizons in Generative AI: Science, Creativity, and Society.

Our conference today on new horizons in generative AI, invites us to think of the frontier of research and innovation.

These stories will expose some features of the sociotechnical foundations of generative AI that is my underlying message and call to action.

What he doesn’t know yet, is that this book will become a cornerstone of the field and industry of weather forecasting.

There is a specific and firm model for responsibility that is built on taking a sociotechnical approach to generative AI and our work.

Abstract: This talk talk has a single objective: to advocate for machine learning infused with social purpose.

In this way, social purpose transforms our field of machine learning: into something that is both technical and social.

So social purpose will make machine learning something that is both technical and social.

A more global field and industry can shift machine learning to be more general: magnifying social purpose.

Your continued volunteering, funding, support, and openness to these groups shows yet another way of infusing machine learning with social purpose.

At Youtube, the relationships between system parameters and metrics often seem simple — straight-line models sometimes fit our data well.

To find optimal values of two parameters experimentally, the obvious strategy would be to experiment with and update them in separate, sequential stages.

We address uncertainty in estimates $\widehat \beta$ of model coefficients with subsampling, our usual strategy for variance estimation in online experiments[12].

If our experiments were smaller then we might need to model our metrics with Poisson, Binomial or other GLMs instead.

We sometimes use $L^1$ regularization when we fit models to our live experiment data to select the most important non-linear c…

When measuring an attribute, we must assume that the thing we are trying to measure exists, and then validity tells us if we’re actually measuring it.

Reliability sets the upper bound on your validity, and you cannot measure something validly if you cannot measure it reliably.

Two fundamental concepts emerged from this research: reliability and validity.For any procedure, reliability is necessary for high-quality data but not sufficient.

Inference for non-parametric reliability and validity measurementsThere are derivations for standard errors and confidence intervals for some non-parametric reliability metrics.

In practice, we measure reliability with both non-parametric and parametric met…

Both of these assumptions seem reasonable and general and we can quickly see where Gaussian distribution should start popping out.

Since Gaussian distribution is fully defined by just two numbers - mean and variance, we can, by collecting enough data rather precisely estimate these values.

Below I simulate 400x200=80000 independent pixels, each taking a random value between zero and one [code available here].

These distributions we often refer to as "fat tails" [for putting much more "weight" in the tail of the distribution, as in outside of the "central" part] are really weird and problematic.

Let's get back to why we see Gaussian distribution anywhere at all?

So it should not be surprising that hubris in science, displayed by some scientists strikes me equally as arrogant and irritating as the claims of religious leaders.

With the day of judgment established and imminent we are now ready to spread the gospel of modern day religion and profit.

AGI (artificial general intelligence) isn't even a well defined thing.

We can hardly even imagine such a thing, which is great for religious purposes, because everyone can have their own "AGI" to be scared of.

That we should never question the technocrats, because technology is the only and natural way of making our lives better.

NeurIPS 2023, arguably this year’s biggest AI conference takes place in two weeks from Dec 10–16 in New Orleans.

Here are the main trends I observed:Most NLP work at NeurIPS is related to large language models (LLMs).

Papers studied different aspects of in-context learning such as whether it persists during training and using a Bayesian perspective.

Lexinvariant Language Models (Huang et al.).

Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning (Wang et al.).

Sparked by progress in Large Language Models (LLMs), there’s a lot of chatter recently about AGI, its timelines, and what it might look like.

In particular, when I am asked about what I think AGI will look like, I personally like to point to self-driving.

AGISo these are the broad strokes of what I think AGI will look like.

And it looks a lot more like self-driving, the part of our economy that is currently speed-running the development of a major, society-altering automation.

I personally very much look forward to what all the equivalents of that might be with AGI.

Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector.

Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs.

However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs.

Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information.

Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning…

Limited diversity in standardized benchmarks for evaluating audio representation learning (ARL) methods may hinder systematic comparison of current methods' capabilities.

We present ARCH, a comprehensive benchmark for evaluating ARL methods on diverse audio classification domains, covering acoustic events, music, and speech.

ARCH streamlines benchmarking of ARL techniques through its unified access to a wide range of domains and its ability to readily incorporate new datasets and models.

To address the current lack of open-source, pre-trained models for non-speech audio, we also release new pre-trained models that demonstrate strong performance on non-speech datasets.

We argue that the pres…

Practical Bayesian learning often requires (1) online inference, (2) dynamic models, and (3) ensembling over multiple different models.

Recent advances have shown how to use random feature approximations to achieve scalable, online ensembling of Gaussian processes with desirable theoretical properties and fruitful applications.

One key to these methods' success is the inclusion of a random walk on the model parameters, which makes models dynamic.

We show that these methods can be generalized easily to any basis expansion model and that using alternative basis expansions, such as Hilbert space Gaussian processes, often results in better performance.

Finally, we propose a novel method to ense…

Bayesian few-shot classification has been a focal point in the field of few-shot learning.

This paper seamlessly integrates mirror descent-based variational inference into Gaussian process-based few-shot classification, addressing the challenge of non-conjugate inference.

By leveraging non-Euclidean geometry, mirror descent achieves accelerated convergence by providing the steepest descent direction along the corresponding manifold.

It also exhibits the parameterization invariance property concerning the variational distribution.

Experimental results demonstrate competitive classification accuracy, improved uncertainty quantification, and faster convergence compared to baseline models.

For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge.

In this paper, we propose a new way of self-attention calculation, termed Consistent Self-Attention, that significantly boosts the consistency between the generated images and augments prevalent pretrained diffusion-based text-to-image models in a zero-shot manner.

To extend our method to long-range video generation, we further introduce a novel semantic space temporal motion prediction module, named Semantic Motion Predictor.

This module converts the generated sequence of images into vid…

Multilingual information retrieval (MLIR) considers the problem of ranking documents in several languages for a query expressed in a language that may differ from any of those languages.

Recent work has observed that approaches such as combining ranked lists representing a single document language each or using multilingual pretrained language models demonstrate a preference for one language over others.

This work proposes a language fairness metric to evaluate whether documents across different languages are fairly ranked through statistical equivalence testing using the Kruskal-Wallis test.

Thus our proposed measure, PEER (Probability of EqualExpected Rank), is the first fairness metric s…

Large-scale Text-to-Image (T2I) diffusion models demonstrate significant generation capabilities based on textual prompts.

Based on the T2I diffusion models, text-guided image editing research aims to empower users to manipulate generated images by altering the text prompts.

However, existing image editing techniques are prone to editing over unintentional regions that are beyond the intended target area, primarily due to inaccuracies in cross-attention maps.

To address this problem, we propose Localization-aware Inversion (LocInv), which exploits segmentation maps or bounding boxes as extra localization priors to refine the cross-attention maps in the denoising phases of the diffusion proc…

However, the unpredictable conditions of space pose significant challenges for robotic systems, necessitating the development of advanced learning techniques to enable autonomous assembly.

In this study, we present a novel approach for learning autonomous peg-in-hole assembly in the context of space robotics.

Our focus is on enhancing the generalization and adaptability of autonomous systems through deep reinforcement learning.

We demonstrate the adaptability of our agents to novel scenarios and assembly sequences while emphasizing the potential of leveraging advanced simulation techniques for robot learning in space.

Our findings set the stage for future advancements in intelligent robotic…

Learning to solve vehicle routing problems (VRPs) has garnered much attention.

In this paper, we aim to develop a unified neural solver that can cope with a range of VRP variants simultaneously.

Specifically, we propose a multi-task vehicle routing solver with mixture-of-experts (MVMoE), which greatly enhances the model capacity without a proportional increase in computation.

We further develop a hierarchical gating mechanism for the MVMoE, delivering a good trade-off between empirical performance and computational complexity.

Experimentally, our method significantly promotes the zero-shot generalization performance on 10 unseen VRP variants, and showcases decent results on the few-shot set…

Traditionally, natural language processing (NLP) models often use a rich set of features created by linguistic expertise, such as semantic representations.

However, in the era of large language models (LLMs), more and more tasks are turned into generic, end-to-end sequence generation problems.

In this paper, we investigate the question: what is the role of semantic representations in the era of LLMs?

Specifically, we investigate the effect of Abstract Meaning Representation (AMR) across five diverse NLP tasks.

We recommend focusing on these areas for future work in semantic representations for LLMs.

The Pareto front of a set of vectors is the subset which is comprised solely of all of the best trade-off points.

In this work, we prove a very useful result which states that all Pareto front surfaces can be explicitly parametrised using polar coordinates.

Consequently, by exploiting this representation, we show how it is possible to generalise many useful concepts from linear algebra, probability and statistics, and decision theory to function over the space of Pareto front surfaces.

Notably, we focus our attention on the stochastic setting where the Pareto front surface itself is a stochastic process.

Besides this, we also illustrate how these Pareto front ideas can be used within the co…

Graph augmentation has received great attention in recent years for graph contrastive learning (GCL) to learn well-generalized node/graph representations.

However, mainstream GCL methods often favor randomly disrupting graphs for augmentation, which shows limited generalization and inevitably leads to the corruption of high-level graph information, i.e., the graph community.

Moreover, current knowledge-based graph augmentation methods can only focus on either topology or node features, causing the model to lack robustness against various types of noise.

To address these limitations, this research investigated the role of the graph community in graph augmentation and figured out its crucial …

Argument structure learning~(ASL) entails predicting relations between arguments.

Because it can structure a document to facilitate its understanding, it has been widely applied in many fields~(medical, commercial, and scientific domains).

Despite its broad utilization, ASL remains a challenging task because it involves examining the complex relationships between the sentences in a potentially unstructured discourse.

To resolve this problem, we have developed a simple yet effective approach called Dual-tower Multi-scale cOnvolution neural Network~(DMON) for the ASL task.

Specifically, we organize arguments into a relationship matrix that together with the argument embeddings forms a relatio…

Change detection as an interdisciplinary discipline in the field of computer vision and remote sensing at present has been receiving extensive attention and research.

We propose MFDS-Net: Multi-Scale Feature Depth-Supervised Network for Remote Sensing Change Detection with Global Semantic and Detail Information (MFDS-Net) with the aim of achieving a more refined description of changing buildings as well as geographic information, enhancing the localisation of changing targets and the acquisition of weak features.

We propose the Global Semantic Enhancement Module (GSEM) to enhance the processing of high-level semantic information from a global perspective.

The Differential Feature Integratio…

Session-based recommendation (SBR) aims to predict the following item a user will interact with during an ongoing session.

Most existing SBR models focus on designing sophisticated neural-based encoders to learn a session representation, capturing the relationship among session items.

However, they tend to focus on the last item, neglecting diverse user intents that may exist within a session.

To address this issue, we propose a novel SBR model, called Multi-intent-aware Session-based Recommendation Model (MiaSRec).

MiaSRec represents various user intents by deriving multiple session representations centered on each item and dynamically selecting the important ones.

Integrating the two most common types of sensory data, images, and point clouds, can enhance detection accuracy.

However, currently, no model exists that can simultaneously detect an object's position in both point clouds and images and ascertain their corresponding relationship.

Furthermore, to assess the accuracy of the object correlation between point clouds and images, this paper proposes a new evaluation metric, Consistency Precision (CP).

The study also explored how the proposed consistency detection method performs on images when the calibration parameters between images and point clouds are disturbed, compared to existing post-processing methods.

The experimental results demonstrate…

Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe.

We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to using hundreds of GPUs for training.

NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN).

Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting.

NeMo-Aligner is designed for extensibility, allowing support for other alignment…

This paper introduces UQA, a novel dataset for question answering and text comprehension in Urdu, a low-resource language with over 70 million native speakers.

UQA is generated by translating the Stanford Question Answering Dataset (SQuAD2.0), a large-scale English QA dataset, using a technique called EATS (Enclose to Anchor, Translate, Seek), which preserves the answer spans in the translated context paragraphs.

The paper also benchmarks several state-of-the-art multilingual QA models on UQA, including mBERT, XLM-RoBERTa, and mT5, and reports promising results.

UQA is a valuable resource for developing and testing multilingual NLP systems for Urdu and for enhancing the cross-lingual transf…

Large Language Models (LLMs) have achieved remarkable success across diverse tasks, yet they remain vulnerable to adversarial attacks, notably the well-documented \textit{jailbreak} attack.

Recently, the Greedy Coordinate Gradient (GCG) attack has demonstrated efficacy in exploiting this vulnerability by optimizing adversarial prompts through a combination of gradient heuristics and greedy search.

However, the efficiency of this attack has become a bottleneck in the attacking process.

Specifically, we introduce the \textbf{M}omentum \textbf{A}ccelerated G\textbf{C}G (\textbf{MAC}) attack, which incorporates a momentum term into the gradient heuristic.

Experimental results showcase the notab…

Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs.

However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations.

To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements.

Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria.

On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and…

Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness.

Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values.

We propose Stein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency.

Specifically, S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy.

Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC …

Unlearnable examples (UEs) seek to maximize testing error by making subtle modifications to training examples that are correctly labeled.

The other approach is pre-training purification, e.g., image short squeezing, which consists of several simple compressions but often encounters challenges in dealing with various UEs.

Our work provides a novel disentanglement mechanism to build an efficient pre-training purification method.

Firstly, we uncover rate-constrained variational autoencoders (VAEs), demonstrating a clear tendency to suppress the perturbations in UEs.

Based on this network, a two-stage purification approach is naturally developed.

Different from the context-independent (CI) concepts such as human, car, and airplane, context-dependent (CD) concepts require higher visual understanding ability, such as camouflaged object and medical lesion.

Despite the rapid advance of many CD understanding tasks in respective branches, the isolated evolution leads to their limited cross-domain generalisation and repetitive technique innovation.

This restricts their real-world CD concept understanding towards artificial general intelligence (AGI).

We propose a unified model with a single set of parameters, Spider, which only needs to be trained once.

With the help of the proposed concept filter driven by the image-mask group prompt, Spi…

Deep learning models often encounter challenges in making accurate inferences when there are domain shifts between the source and target data.

This issue is particularly pronounced in clinical settings due to the scarcity of annotated data resulting from the professional and private nature of medical data.

To tackle domain shifts in data-scarce medical scenarios, we propose a Random frequency filtering enabled Single-source Domain Generalization algorithm (RaffeSDG), which promises robust out-of-domain inference with segmentation models trained on a single-source domain.

A filter-based data augmentation strategy is first proposed to promote domain variability within a single-source domain b…

Simulating soil reflectance spectra is invaluable for soil-plant radiative modeling and training machine learning models, yet it is difficult as the intricate relationships between soil structure and its constituents.

To address this, a fully data-driven soil optics generative model (SOGM) for simulation of soil reflectance spectra based on soil property inputs was developed.

It generates soil reflectance spectra from text-based inputs describing soil properties and their values rather than only numerical values and labels in binary vector format.

The generative model can simulate output spectra based on an incomplete set of input properties.

The testing results of the SOGM on new datasets …

This paper introduces SparseTSF, a novel, extremely lightweight model for Long-term Time Series Forecasting (LTSF), designed to address the challenges of modeling complex temporal dependencies over extended horizons with minimal computational resources.

At the heart of SparseTSF lies the Cross-Period Sparse Forecasting technique, which simplifies the forecasting task by decoupling the periodicity and trend in time series data.

This technique involves downsampling the original sequences to focus on cross-period trend prediction, effectively extracting periodic features while minimizing the model's complexity and parameter count.

Based on this technique, the SparseTSF model uses fewer than 1k…

This report presents the ECO (Ensembled Clip score and cOnsensus score) pipeline from team DSBA LAB, which is a new framework used to evaluate and rank captions for a given image.

It is made possible by combining an Ensembled CLIP score, which considers the semantic alignment between the image and captions, with a Consensus score that accounts for the essentialness of the captions.

Using this framework, we achieved notable success in the CVPR 2024 Workshop Challenge on Caption Re-ranking Evaluation at the New Frontiers for Zero-Shot Image Captioning Evaluation (NICE).

Specifically, we secured third place based on the CIDEr metric, second in both the SPICE and METEOR metrics, and first in th…

Open-vocabulary Temporal Action Detection (Open-vocab TAD) is an advanced video analysis approach that expands Closed-vocabulary Temporal Action Detection (Closed-vocab TAD) capabilities.

Closed-vocab TAD is typically confined to localizing and classifying actions based on a predefined set of categories.

In contrast, Open-vocab TAD goes further and is not limited to these predefined categories.

The prevalent methods in Open-vocab TAD typically employ a 2-stage approach, which involves generating action proposals and then identifying those actions.

Additionally, existing studies face challenges in handling actions of different durations owing to the use of fixed temporal processing methods.

This study addresses the challenge of detecting semantic column types in relational tables, a key task in many real-world applications.

While language models like BERT have improved prediction accuracy, their token input constraints limit the simultaneous processing of intra-table and inter-table information.

We propose a novel approach using Graph Neural Networks (GNNs) to model intra-table dependencies, allowing language models to focus on inter-table information.

Our proposed method not only outperforms existing state-of-the-art algorithms but also offers novel insights into the utility and functionality of various GNN types for semantic type detection.

The code is available at https://g…

Responsibility & Safety The ethics of advanced AI assistants ShareCopy link ×Exploring the promise and risks of a future with more capable AI Imagine a future where we interact regularly with a range of advanced artificial intelligence (AI) assistants — and where millions of assistants interact with each other on our behalf.

General-purpose foundation models are paving the way for increasingly advanced AI assistants.

Advanced AI assistants could have a profound impact on users and society, and be integrated into most aspects of people’s lives.

Able to fluidly communicate using natural language, the written output and voices of advanced AI assistants may become hard to distinguish from those…

Research TacticAI: an AI assistant for football tactics ShareCopy link ×As part of our multi-year collaboration with Liverpool FC, we develop a full AI system that can advise coaches on corner kicks 'Corner taken quickly… Origi!'

Our first paper, Game Plan, looked at why AI should be used in assisting football tactics, highlighting examples such as analyzing penalty kicks.

Predicting corner kick outcomes with geometric deep learning A corner kick is awarded when the ball passes over the byline, after touching a player of the defending team.

With TacticAI, we have developed a capable AI assistant for football tactics and achieved a milestone in developing useful assistants in sports AI.

We s…

In a new technical report, we introduce SIMA, short for Scalable Instructable Multiworld Agent, a generalist AI agent for 3D virtual settings.

SIMA: a versatile AI agent SIMA is an AI agent that can perceive and understand a variety of environments, then take actions to achieve an instructed goal.

What’s more, an agent trained in all but one game performed nearly as well on that unseen game as an agent trained specifically on it, on average.

We compare this performance with three types of generalist SIMA agent, each trained across multiple environments.

Advancing AI agent research SIMA’s results show the potential to develop a new wave of generalist, language-driven AI agents.

Today, we’re excited to introduce a new generation of open models from Google to assist developers and researchers in building AI responsibly.

Gemma open modelsGemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models.

This enables Gemma 2B and 7B to achieve best-in-class performance for their sizes compared to other open models.

And Gemma models are capable of running directly on a developer laptop or desktop computer.

Notably, Gemma surpasses significantly larger models on key benchmarks while adhering to our rigorous standards for safe and responsible outputs.

A note from Google and Alphabet CEO Sundar Pichai:Last week, we rolled out our most capable model, Gemini 1.0 Ultra, and took a significant step forward in making Google products more helpful, starting with Gemini Advanced.

Today, developers and Cloud customers can begin building with 1.0 Ultra too — with our Gemini API in AI Studio and in Vertex AI.

Our teams continue pushing the frontiers of our latest models with safety at the core.

In fact, we’re ready to introduce the next generation: Gemini 1.5.

It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute.

We’re excited by the progress, for example with our Search Generative Experience, or SGE, which you can try in Search Labs.

Introducing Gemini AdvancedBard has been the best way for people to directly experience our most capable models.

To reflect the advanced tech at its core, Bard will now simply be called Gemini.

The version with Ultra will be called Gemini Advanced, a new experience far more capable at reasoning, following instructions, coding, and creative collaboration.

You can start using Gemini Advanced by subscribing to the new Google One AI Premium plan, which offers the best of Google’s AI features in a single place.

Research AlphaGeometry: An Olympiad-level AI system for geometry ShareCopy link ×Our AI system surpasses the state-of-the-art approach for geometry problems, advancing AI reasoning in mathematicsReflecting the Olympic spirit of ancient Greece, the International Mathematical Olympiad is a modern-day arena for the world's brightest high-school mathematicians.

In a paper published today in Nature, we introduce AlphaGeometry, an AI system that solves complex geometry problems at a level approaching a human Olympiad gold-medalist - a breakthrough in AI performance.

In a benchmarking test of 30 Olympiad geometry problems, AlphaGeometry solved 25 within the standard Olympiad time limit.

Solving Ol…

Today we’re announcing a suite of advances in robotics research that bring us a step closer to this future.

AutoRT, SARA-RT, and RT-Trajectory build on our historic Robotics Transformers work to help robots make decisions faster, and better understand and navigate their environments.

So the AutoRT system comprises layers of practical safety measures from classical robotics.

SARA-RT: Making Robotics Transformers leaner and faster Our new system, Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT), converts Robotics Transformer (RT) models into more efficient versions.

We will continue to tackle challenges in robotics today and to adapt to the new capabilities and technologies …

Research Images altered to trick machine vision can influence humans too ShareCopy link ×New research shows that even subtle changes to digital images, designed to confuse computer vision systems, can also affect human perception Computers and humans see the world in different ways.

Our discovery highlights a similarity between human and machine vision, but also demonstrates the need for further research to understand the influence adversarial images have on people, as well as AI systems.

An adversarial image is one that has been subtly altered by a procedure that causes an AI model to confidently misclassify the image contents.

Indeed, security concerns have led researchers to investigate …

Company 2023: A Year of Groundbreaking Advances in AI and Computing ShareCopy link ×This has been a year of incredible progress in the field of Artificial Intelligence (AI) research and its practical applications.

In this Year-in-Review post we’ll go over some of Google Research’s and Google DeepMind’s efforts putting these paragraphs into practice safely throughout 2023.

In November, in partnership with YouTube, we announced Lyria, our most advanced AI music generation model to date.

Then in December, we launched Gemini, our most capable and general AI model.

Advances in capable language and multimodal models have also benefited our robotics research efforts.

Research FunSearch: Making new discoveries in mathematical sciences using Large Language Models ShareCopy link ×By searching for “functions” written in computer code, FunSearch made the first discoveries in open problems in mathematical sciences using LLMs Large Language Models (LLMs) are useful assistants - they excel at combining concepts and can read, write and code to help people solve problems.

This work represents the first time a new discovery has been made for challenging open problems in science or mathematics using LLMs.

FunSearch discovered new solutions for the cap set problem, a longstanding open problem in mathematics.

The corresponding function produced by FunSearch for each …

We’ll be showcasing demos of our cutting edge AI models for global weather forecasting, materials discovery, and watermarking AI-generated content.

There will also be an opportunity to hear from the team behind Gemini, our largest and most capable AI model.

Here’s a look at some of our research highlights:Multimodality: language, video, actionUniSim is a universal simulator of real-world interactions.

Our approach surpassed current methods on vision and language tasks, and showed more potential to scale.

To help people solve problems and find what they’re looking for, AI models need to process billions of unique values efficiently.

AI has the potential to create opportunities — from the everyday to the extraordinary — for people everywhere.

That’s what excites me: the chance to make AI helpful for everyone, everywhere in the world.

At the same time, developers are using our models and infrastructure to build new generative AI applications, and startups and enterprises around the world are growing with our AI tools.

Now, we’re taking the next step on our journey with Gemini, our most capable and general model yet, with state-of-the-art performance across many leading benchmarks.

I’m genuinely excited for what’s ahead, and for the opportunities Gemini will unlock for people everywhere.

Research Millions of new materials discovered with deep learning ShareCopy link ×AI tool GNoME finds 2.2 million new crystals, including 380,000 stable materials that could power future technologies Modern technologies from computer chips and batteries to solar panels rely on inorganic crystals.

Computational approaches drawing from the Materials Project, Open Quantum Materials Database and WBM database boosted this number to 48,000 stable crystals.

Over the last decade, computational approaches led by the Materials Project and other groups have helped discover 28,000 new materials.

GNoME: Harnessing graph networks for materials explorationGNoME uses two pipelines to discover low-energy (st…

AI Infrastructure for RAGPrior to the rise of Generative AI, a typical application architecture might involve a database, a set of microservices, and a frontend.

Even the most basic RAG applications introduce new requirements for serving LLMs, processing, and retrieving unstructured data.

Many customers choose to access AI infrastructure like TPUs and GPUs via a fully managed platform, such as Vertex AI.

This is why we have developed a quickstart solution and reference architecture for RAG applications built on top of GKE, Cloud SQL, and open-source frameworks Ray, LangChain and Hugging Face.

Benefits of RAG on GKE and Cloud SQLGKE and Cloud SQL accelerate your journey to production in a va…

If reading and constructing a batch takes longer than GPU computation, then the GPU is effectively stalled and underutilized, leading to longer training times.

In Dataflux, we employ a Cloud Storage feature called Compose Objects that can dynamically combine many smaller objects into a larger object.

Another optimization that Dataflux Datasets employs is high-throughput parallel-listing, speeding up the initial metadata needed for the dataset.

Dataflux Dataset uses these client libraries under the hood.

Dataflux is available nowGive the Dataflux Dataset for PyTorch (or the Dataflux Python client library if writing your own ML training dataset code) a try and let us know how it boosts your w…

Early signals tell us that generative AI is that missing catalyst, and Google Cloud is a unique partner for your journey.

Why generative AI catalyzes your teamA business strategy with generative AI at the center benefits customers and employees.

It’s not just about generative AI; it’s about what it takes to be good at generative AI.

There’s more than one way to proceed with your generative AI strategy, but at Google Cloud, we see three crucial building blocks for your success.

Register for our Building Apps in an AI Era webinar to learn more about how Google Cloud can help you innovate faster, deliver unparalleled customer experiences, and secure a lasting competitive advantage.

From automating business processes to enhancing web experiences and improving customer service, the applications of generative AI are endless.

Additionally, Vertex AI Agent Builder streamlines the process of grounding generative AI outputs in enterprise data.

Leading AI companies are building on Vertex AICustomers choose Vertex AI because Google Cloud helps provide the necessary trust for organizations to confidently deploy their services.

Altogether, Vertex AI helps organizations build transformative generative AI experiences with confidence and speed.

Gartner, Magic Quadrant for Cloud AI Developer Services, Jim Scheibmeir, Arun Batchu, Mike Fang - April 29, 2024.

A2 & A3 machine types come with up to 6TB of Local SSD already bundled, which can be used for the Cloud Storage Fuse file cache.

This is useful if you want to limit the total capacity the Cloud Storage FUSE cache can use within its mounted directory.

GKE users are using Cloud Storage FUSE to get simple file access to their objects, through familiar Kubernetes APIs, via the Cloud Storage FUSE CSI.

And now, you can enable the Cloud Storage FUSE file cache in GKE to accelerate performance as well.

This controls the maximum size of the Cloud Storage FUSE file cache.

In this blog, we illustrate how Workflows can perform long-document summarization, a concrete use case with wide applicability.

Open-source LLM orchestration frameworks like LangChain for Python and TypeScript developers, or LangChain4j for Java developers, integrate various components such as LLMs, document loaders, and vector databases, to implement complex tasks such as document summarization.

For each section, a summary is created, and a summary of all the summaries is created as a final step.

Iterative refinement — Similar to the map/reduce approach, we evaluate the document in a piecemeal fashion.

However, the map/reduce approach has one advantage over the refinement method.

BigQuery's integrated speech-to-text functionality offers a powerful tool for unlocking valuable insights hidden within audio data.

BigQuery speech-to-text transforms audio data into actionable insights, offering potential benefits across industries and enabling a deeper understanding of customer interactions across multiple channels.

You can also use BigQuery ML to leverage Gemini 1.0 Pro to gain additional insights & data formatting such as entity extraction and sentiment analysis to the text extracted from audio files using BigQuery ML’s native speech-to-text capability.

BigQuery's ML.TRANSCRIBE function, connected to a pre-trained speech-to-text model hosted on Google's Vertex AI platfo…

Monitoring machine learning (ML) models in production is now as simple as using a function in BigQuery!

Today we're introducing a new set of functions that enable model monitoring directly within BigQuery.

Now, you can describe data throughout the model workflow by profiling training or inference data, monitor skew between training and serving data, and monitor drift in serving data over time using SQL — for BigQuery ML models as well as any model whose feature training and serving data is available through BigQuery.

The foundation of a model: the dataA model is only as good as the data it learns from.

Understanding the data deeply is essential for effective feature engineering, model selec…

For Media customers, generative AI introduces the ability to generate real time, personalized and unique interactions that weren’t possible before.

How generative AI is transforming the telco and media industryThe telecommunications and media industry is at the forefront of integrating generative AI into their operations, viewing it as a catalyst for growth and innovation.

Embedding responsible AI governance throughout our processesAs we work to develop generative AI responsibly, we keep a close eye on emerging regulatory frameworks.

By integrating responsible AI principles and toolkits into all aspects of AI development, we are witnessing a growing confidence among organizations in using G…

We are excited to launch PyTorch/XLA 2.3 this week.

The 2.3 release brings with it even more productivity, performance and usability improvements.

Before we get into the release updates, here’s a short overview of why PyTorch/XLA is great for model training, fine-tuning and serving.

Additionally, autocasting to bf16 has provided crucial flexibility, allowing certain parts of our graph to operate on fp32, optimizing our model’s performance.

- Yoav HaCohen, Research team lead, LightricksWhat's in the 2.3 release: Distributed training, dev experience, and GPUsPyTorch/XLA 2.3 keeps us current with PyTorch Foundation's 2.3 release from earlier this week, and offers notable upgrades from PyTorch/…

For many organizations, developer productivity has been "stuck" for a while now.

The topic of developer productivity came to a head in the second half of 2023, as people debated new ways to measure software delivery performance.

How do you actually measure software developers?

It explores what it means to measure developer productivity and how generative AI will make a difference.

We look at frameworks like DORA and SPACE, offer questions for assessing your own organization’s approach to developer productivity, and then explain how AI-assisted developer tooling improves your productivity outcomes.

At Prewave, we’re driven by the mission to help companies make their entire supply chains more resilient, transparent, and sustainable.

That’s why we built the Prewave supply chain risk intelligence platform on Google Cloud from inception in 2019.

And second, it makes supply chains more sustainable by detecting and solving ESG risks, such as forced labor or environmental issues.

With this information, Prewave also maps our clients’ supply chains from immediate and sub-tier suppliers down to the raw materials’ providers.

We’re confident that our collaboration with Google Cloud will continue to bring us huge benefits as we help more companies internationally to achieve transparency, resilienc…

Google Cloud Champion Innovators are a global network of more than 600 non-Google professionals, who are technical experts in Google Cloud products and services.

I find the Google Cloud developer blog and YouTube channel particularly instructive when it comes to real use cases.

Google Developer Groups, or GDGs, for example are a great way to network and keep up with the latest developments.

Becoming a Google Cloud Champion Innovator has been really beneficial.

Take the next steps on your Google Cloud journey and learn more about the Google Cloud Innovators Program, designed to help developers and practitioners grow their Google Cloud skills and advance their careers.

Next on the horizon is expanding to big data solutions with Ray Data and BigQuery.

With just two DevOps engineers and one MLOps engineer, IPRally was able to build its own customized ML platform with GKE and Ray as key components.

To more easily manage Ray, IPRally uses KubeRay, a specialized tool that simplifies Ray cluster management on Kubernetes.

IPRally uses Ray for the most advanced tasks like massive preprocessing of data and exploratory deep learning in R&D.

And byleveraging them within GKE, IPRally facilitates the creation of nodes on-demand, scales GPU resources as needed, thus optimizing its operational costs.

Customers with a sustained level of tokens per minute (TPM) usage on GPT-4 or GPT-4 Turbo can request access to provisioned throughput to get discounts ranging from 10–50% based on the size of the commitment.

Reduced costs on asynchronous workloads: Customers can use our new Batch API to run non-urgent workloads asynchronously.

Batch API requests are priced at 50% off shared prices, offer much higher rate limits, and return results within 24 hours.

We plan to keep adding new features focused on enterprise-grade security, administrative controls, and cost management.

For more information on these launches, visit our API documentation or get in touch with our team to discuss custom solution…

OpenAI, alongside industry leaders including Amazon, Anthropic, Civitai, Google, Meta, Metaphysic, Microsoft, Mistral AI, and Stability AI, has committed to implementing robust child safety measures in the development, deployment, and maintenance of generative AI technologies as articulated in the Safety by Design principles.

By adopting comprehensive Safety by Design principles, OpenAI and our peers are ensuring that child safety is prioritized at every stage in the development of AI.

Responsibly source our training datasets, detect and remove child sexual abuse material (CSAM) and child sexual exploitation material (CSEM) from training data, and report any confirmed CSAM to the relevant a…

Our new local presence also gets us closer to leading businesses like Daikin, Rakuten, and TOYOTA Connected who are using ChatGPT Enterprise to automate complex business processes, assist in data analysis, and optimize internal reporting.

ChatGPT also helps accelerate the efforts of local governments, such as Yokosuka City, which is leveraging the technology to improve the efficiency of public services in Japan.

Over the past year, the city has gradually provided ChatGPT access to almost all city employees, and 80% have reported increases in productivity.

Now Yokosuka City has formed a network with 21 local governments—including the Tokyo Metropolitan Government and the City of Kobe—to …

Assisted Fine-TuningAt DevDay last November, we announced a Custom Model program designed to train and optimize models for a specific domain, in partnership with a dedicated group of OpenAI researchers.

Since then, we've met with dozens of customers to assess their custom model needs and evolved our program to further maximize performance.

Today, we are formally announcing our assisted fine-tuning offering as part of the Custom Model program.

Fully custom-trained models imbue new knowledge from a specific domain by modifying key steps of the model training process using novel mid-training and post-training techniques.

Our team modified every step of the model training process, from domain-s…

We’ve also introduced additional content safeguards for this experience, such as blocking prompts and generations in a wider range of categories.

There are many benefits to creating an account including the ability to save and review your chat history, share chats, and unlock additional features like voice conversations and custom instructions.

For anyone that has been curious about AI’s potential but didn’t want to go through the steps to set-up an account, start using ChatGPT today.

We recognize that generating speech that resembles people's voices has serious risks, which are especially top of mind in an election year.

We are engaging with U.S. and international partners from across government, media, entertainment, education, civil society and beyond to ensure we are incorporating their feedback as we build.ÂThe partners testing Voice Engine today have agreed to our usage policies, which prohibit the impersonation of another individual or organization without consent or legal right.

In addition, our terms with these partners require explicit and informed consent from the original speaker and we don’t allow developers to build ways for individual users to create the…

Starting his career at DreamWorks Animation, Don Allen III is a multidisciplinary creator, speaker and consultant who collaborates with major tech and entertainment companies on mixed reality, virtual reality and AI applications.

“For a long time I've been making augmented reality hybrid creatures that I think would be fun combinations in my head.

Now I have a much easier way of prototyping the ideas before I fully build out the 3-D characters to place in spatial computers.” Don cites Sora’s “weirdness” as its greatest strength: “It’s not bound by traditional laws of physics or conventions of thought.” He says that working with Sora shifted his focus from “technical hurdle…

Echoing this sentiment, Louis Dreyfus, CEO of Le Monde, stated, "At the moment we are celebrating the 80th anniversary of Le Monde, this partnership with OpenAI allows us to expand our reach and uphold our commitment to providing accurate, verified, balanced news stories at scale.

Collaborating with OpenAI ensures that our authoritative content can be accessed and appreciated by a broader, more diverse audience. ÂEvery shift in the media landscape has presented Le Monde with new opportunities.

From the transition to digital platforms to embracing the era of free media, Le Monde has consistently seized these moments to underscore its commitment to independence, expertise, and journalistic i…

Additionally, Sam Altman, CEO, will rejoin the OpenAI Board of Directors.ÂSue, Nicole and Fidji have experience in leading global organizations and navigating complex regulatory environments, including backgrounds in technology, nonprofit and board governance.

They will work closely with current board members Adam D’Angelo, Larry Summers and Bret Taylor as well as Sam and OpenAI’s senior management.ÂBret Taylor, Chair of the OpenAI board, stated, “I am excited to welcome Sue, Nicole, and Fidji to the OpenAI Board of Directors.

She also served as President of Sony Entertainment, Inc., and simultaneously served as President of Sony Corporation of America.

She also serves as a member of …

The Special Committee of the OpenAI Board today announced the completion of the review by WilmerHale.

The firm conducted dozens of interviews with members of OpenAI’s prior Board, OpenAI executives, advisors to the prior Board, and other pertinent witnesses; reviewed more than 30,000 documents; and evaluated various corporate actions.

“We have unanimously concluded that Sam and Greg are the right leaders for OpenAI,” stated Bret Taylor, Chair of the OpenAI Board.

The Special Committee acknowledged the important work done by WilmerHale in conducting this extensive review and thanked OpenAI current and former Board members, advisors and employees for their cooperation.

The Special Commi…

Date: January 31, 2018 at 11:54:30 PM PSTSubject: Re: Top AI institutions todayWorking at the cutting edge of AI is unfortunately expensive.

For example,In addition to DeepMind, Google also has Google Brain, Research, and Cloud.

If historical trends are any indication, progress in AI is primarily driven by systems - compute, data, infrastructure.

Not only that, but any algorithmic advances published in a paper somewhere can be almost immediately re-implemented and incorporated.

The “second stage” would be a full self driving solution based on large-scale neural network training, which OpenAI expertise could significantly help accelerate.

This technical report focuses on (1) our method for turning visual data of all types into a unified representation that enables large-scale training of generative models, and (2) qualitative evaluation of Sora’s capabilities and limitations.

Model and implementation details are not included in this report.

Much prior work has studied generative modeling of video data using a variety of methods, including recurrent networks,[^1][^2][^3] generative adversarial networks,[^4][^5][^6][^7] autoregressive transformers,[^8][^9] and diffusion models.

[^10][^11][^12] These works often focus on a narrow category of visual data, on shorter videos, or on videos of a fixed size.

Sora is a generalist mo…

Based on collaboration and information sharing with Microsoft, we disrupted five state-affiliated malicious actors: two China-affiliated threat actors known as Charcoal Typhoon and Salmon Typhoon; the Iran-affiliated threat actor known as Crimson Sandstorm; the North Korea-affiliated actor known as Emerald Sleet; and the Russia-affiliated actor known as Forest Blizzard.

The identified OpenAI accounts associated with these actors were terminated.

Salmon Typhoon used our services to translate technical papers, retrieve publicly available information on multiple intelligence agencies and regional threat actors, assist with coding, and research common ways processes could be hidden on a system.…

We’re testing memory with ChatGPT.

Remembering things you discuss across all chats saves you from having to repeat information and makes future conversations more helpful.

You're in control of ChatGPT's memory.

You can explicitly tell it to remember something, ask it what it remembers, and tell it to forget conversationally or through settings.

We are rolling out to a small portion of ChatGPT free and Plus users this week to learn how useful it is.

NEW RESEARCHCan Large Language Models Transform Natural Language Intent into Formal Method Postconditions?

However, there is no guarantee that a program’s implementation aligns with its natural language documentation.

However, this information is often underutilized, due to the inherent ambiguity of natural language which makes natural language intent challenging to check programmatically.

The “emergent abilities” of large language models (LLMs) have the potential to facilitate the translation of natural language intent to programmatically checkable assertions.

In a new paper: Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?

Modern computer systems and applications, with unprecedented scale, complexity, and security needs, require careful co-design and co-evolution of hardware and software.

We are pleased to share that eight papers from Microsoft researchers and their collaborators have been accepted to the conference, spanning a broad spectrum of topics.

Regarding infrastructure, topics include memory safety with CHERI, I/O prefetching in modern storage, and smart oversubscription of burstable virtual machines.

Burstable virtual machines (BVMs) are a type of virtual machine in the cloud that allows temporary increases in resource allocation.

We are always pushing the boundaries of computer systems to improve t…

What would it take to build an interactive AI system that could assist you with any task in the physical world, just as a real-time expert would?

To begin exploring the core competencies that such a system would require, we developed and released the Situated Interactive Guidance, Monitoring, and Assistance (SIGMA) system, an open-source research platform and testbed prototype (opens in new tab) for studying mixed-reality task assistance.

SIGMA provides a basis for researchers to explore, understand, and develop the capabilities required to enable in-stream task assistance in the physical world.

Physical and social intelligenceFor AI systems to fluidly collaborate with people in the physica…

Well, I’ve heard other people on your teams use words like surprise, sometimes even shock …HOSN: Yeah, yeah, there are a lot of “wow” factors.

HUIZINGA: Yeah, yeah.

AI research is moving at such speeds that I feel like we need to get accustomed to a timing of now.

HOSN: That’s right.

Well, as we close, Rafah, I want to ask a question anchored on the big idea behind AI Frontiers.

New generations of language models like GPT-4 and Mixtral 8x7B advance the capability to process long input texts.

The first structure, the task description, remains static and independent of the input as a result of conventional prompt optimization techniques.

Despite previous efforts in prompt optimization, the evolution towards more complex prompt structures has rendered many older strategies ineffective in this new context.

SAMMO: A prompt optimization approachDownload SAMMOTo address these challenges, we developed the Structure-Aware Multi-objective Metaprompt Optimization (SAMMO) framework.

We compared it against Automatic Prompt Optimization (APO) and GrIPS, applying open-source mode…

Welcome to Research Focus, a series of blog posts that highlights notable publications, events, code/datasets, new hires and other milestones from across the research community at Microsoft.

NEW RESEARCHAppropriate reliance on Generative AI: Research synthesisAppropriate reliance on AI happens when people accept correct AI outputs and reject incorrect ones.

Spotlight: Event Series Microsoft Research Forum Join us for a continuous exchange of ideas about research in the era of general AI.

They also develop AFRICOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLMR) to create state-of-the-…

Networked systems and their applications are essential in building the reliable, scalable, secure, and innovative infrastructure required to meet society’s evolving needs.

Microsoft is honored to support NDSI ‘24 as a returning sponsor.

We are pleased to announce that 23 papers from Microsoft researchers and their partners have been accepted to the conference.

They encompass both early-stage research and systems already deployed in production.

Spotlight: Microsoft research newsletter Microsoft Research Newsletter Stay connected to the research community at Microsoft.

GRETCHEN HUIZINGA: Welcome to Abstracts, a Microsoft Research Podcast that puts the spotlight on world-class research in brief.

CHAKRABORTY: So satellite connectivity is nothing new and has been there for long.

So we are talking about the satellites that are at least 10 to 20 times cheaper and smaller than state-of-the-art satellites.

So the device sends some packet to the satellite; satellite sends some packet to the device—it’s all about packet exchange.

So our vision is clear: to bring 24-7 connectivity for devices anywhere on Earth with just a click of power button.

The new series “Ideas” debuts with guest Kalika Bali. The speech and language tech researcher talks sci-fi and its impact on her career, the design thinking philosophy behind her research, and the “outrageous idea” she had to work with low-resource languages. The post Ideas: Language technologies for everyone with Kalika Bali appeared first on Microsoft Research.

Welcome to Research Focus, a series of blog posts that highlights notable publications, events, code/datasets, new hires and other milestones from across the research community at Microsoft.

Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools.

However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored.

NEW RESEARCHTraining Audio Captioning Models without AudioAutomated Audio Captioning (AAC) is a process that creates text descriptions for audio recordings.

Their approach leverages CLAP, a contrastive learning model that uses audio an…

And so I just want to start here: for you, Ida, what is general intelligence?

Different people at different times provide different criteria for what would be the artificial general intelligence notion.

One is artificial general intelligence and the other is humanlike intelligence or human-level intelligence.

Artificial general intelligence and humanlike, human-level intelligence—how do these two concepts relate to you?

LLORENS: So it sounds like a very extensive set of experiments across many different tasks and with many different leading AI models, and you’ve uncovered a lack of robustness across some of these different tasks.

AI systems like Bing and Microsoft Copilot (web) are as good as they are because they continuously learn and improve from people’s interactions.

[1]How are people using Copilot (web)?

Conversely, they express explicit frustration or switch topics when encountering mistakes in the response from Copilot (web).

These three reports present a comprehensive and multi-faceted approach to dynamically learning from conversation logs in Copilot (web) at scale.

[1] The research was performed only on fully de-identified interaction data from Copilot (web) consumers.

GRETCHEN HUIZINGA: Welcome to Abstracts, a Microsoft Research Podcast that puts the spotlight on world-class research in brief.

These two examples are also the differences from other deep learning OFDFT works.

This is the generalization challenge and is one of the major challenges of deep learning method for molecular science applications.

This somehow shows the benefits of leveraging the OFDFT framework for using a deep learning method to solve molecular tasks.

You can also read it on arXiv, or you can check out the March 2024 issue of Nature Computational Science.

Welcome to Research Focus, a series of blog posts that highlights notable publications, events, code/datasets, new hires and other milestones from across the research community at Microsoft.

NEW RESEARCHFewer is More: Boosting LLM Reasoning with Reinforced Context PruningLarge language models (LLMs) have shown impressive capabilities, yet they still struggle with math reasoning.

Given that adding more concise CoT examples in the prompt can improve LLM reasoning performance, CoT-Influx employs a coarse-to-fine pruner to maximize the input of effective and concise CoT examples.

The pruner first selects as many crucial CoT examples as possible and then prunes unimportant tokens to fit the cont…

Microsoft cloud monitor platformsWhen they are properly configured, cloud monitors can help to meet monitoring requirements.

Notably, missing monitors and alerts constituted over 40 percent of all misdetections, indicating the complexity of determining what to monitor in cloud services.

Data-driven intelligent monitoringOrganizing monitor dataBecause there is no standardized approach to building monitors, monitor data often lacks structure.

In our paper, “Intelligent Monitoring Framework for Cloud Services: A Data-Driven Approach (opens in new tab),” to be presented at ICSE 2024 (opens in new tab), we propose a data-driven approach for developing this ontology.

This shows us that we can pre…

Luckily, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers have found a treasure trove of abstractions within natural language.

Still, their work represents a step forward for how language models can facilitate increasingly elaborate coding activities.

The method trains on potential tasks and their natural language descriptions, then a language model proposes action abstractions from this dataset.

Just like LILO and Ada, LGA has a novel focus on how natural language leads us to those better abstractions.

These recent papers demonstrate a compelling way forward by placing large language models in an interactive loop with symbolic search, compression, and plannin…

Within the corpus of weather radar data, tornadoes are extremely rare events.

They were particularly eager to apply deep learning, a form of machine learning that excels at processing visual data.

Fusing multiple types of data could improve the accuracy of machine learning models.

Taking steps toward operationsOn top of detecting tornadoes, Kurdzo hopes that models might help unravel the science of why they form.

“I think the forecaster community is still, understandably, skeptical of machine learning.

When introducing new teaching and learning technologies, panelists stressed the importance of iteration and teaching students how to develop critical thinking skills while leveraging technologies like generative AI.

Incorporating generative AI into learning experiencesMIT faculty and instructors aren’t just willing to experiment with generative AI — some believe it’s a necessary tool to prepare students to be competitive in the workforce.

But, she saw an opportunity for teaching experimentation with generative AI.

When incorporating generative AI into assignments, the key is to be clear about learning goals and open to sharing examples of how generative AI could be used in ways that align w…

Slater Professor in Aeronautics and Astronautics, has been named the new head of the Department of Aeronautics and Astronautics (AeroAstro), effective May 1.

She currently directs the Interactive Robotics Group in MIT’s Computer Science and Artificial Intelligence Lab (CSAIL), and MIT’s Industrial Performance Center.

Shah and her team at the Interactive Robotics Group conduct research that aims to imagine the future of work by designing collaborative robot teammates that enhance human capability.

Shah was also named a Bisplinghoff Faculty Fellow, was named to MIT Technology Review’s TR35 List, and received an NSF Faculty Early Career Development Award.

Shah succeeds Professor Steven Barrett…

To do this, they set out to map the spatio-temporal brain dynamics involved in recognizing a visual image.

What they found was that a more distributed network of brain regions than previously thought are actively involved in the encoding and retention processes that underpin memorability.

“We've identified a brain signature of visual memorability that emerges around 300 milliseconds after seeing an image, involving areas across the ventral occipital cortex and temporal cortex, which processes information like color perception and object recognition.

They created a “representational matrix,” which is like a detailed chart, showing how similar neural responses are in various brain regions.

Wh…

Their chip can keep a user’s health records, financial information, or other sensitive data private while still enabling huge AI models to run efficiently on devices.

A digital IMC chip performs computations inside a device’s memory, where pieces of a machine-learning model are stored after being moved over from a central server.

In a side-channel attack, a hacker monitors the chip’s power consumption and uses statistical techniques to reverse-engineer data as the chip computes.

First, they employed a security measure where data in the IMC are split into random pieces.

Safety testingTo test their chip, the researchers took on the role of hackers and tried to steal secret information using s…

To build AI systems that can collaborate effectively with humans, it helps to have a good model of human behavior to start with.

Ultimately, this work could help scientists teach AI systems how humans behave, which could enable these systems to respond better to their human collaborators.

Being able to model human behavior is an important step toward building an AI agent that can actually help that human,” he says.

Modeling behaviorResearchers have been building computational models of human behavior for decades.

Moreover, the researchers saw that their model of human behavior matched up well with measures of player skill (in chess matches) and task difficulty.

Like land-based farming, shellfish aquaculture requires healthy seed production in order to maintain a sustainable industry.

Aquaculture hatchery production of shellfish larvae — seeds — requires close monitoring to track mortality rates and assess health from the earliest stages of life.

Careful observation is necessary to inform production scheduling, determine effects of naturally occurring harmful bacteria, and ensure sustainable seed production.

ARC faces challenges with manually quantifying larvae classes, an important step in their seed production process.

Developing an automated identification and counting system will help to improve this step in the production process with time and…

Although the troposphere is often thought of as the closest layer of the atmosphere to the Earth’s surface, the planetary boundary layer (PBL) — the lowest layer of the troposphere — is actually the part that most significantly influences weather near the surface.

This PBL-focused research effort builds on more than a decade of related work on fast, operational neural network algorithms developed by Lincoln Laboratory for NASA missions.

According to a Global Drought Snapshot report released last year, droughts are a pressing planetary issue that the global community needs to address.

Lack of humidity near the surface, specifically at the level of the PBL, is the leading indicator of drought…

Q: What is the vehicle routing problem, and how do traditional operations research (OR) methods address it?

A: The vehicle routing problem is faced by pretty much every logistics and delivery company like USPS, Amazon, UPS, FedEx, DHL every single day.

To solve the vehicle routing problem, we obviously we can't do our modeling without proper demand information and, ideally, customer-related characteristics.

Then there are a bunch of other equations that define the inner workings of a routing problem.

Q: You’re currently applying machine learning to the vehicle routing problem.

On Vassar Street, in the heart of MIT’s campus, the MIT Stephen A. Schwarzman College of Computing recently opened the doors to its new headquarters in Building 45.

“The college has a broad mandate for computing across MIT,” says Daniel Huttenlocher, dean of the MIT Schwarzman College of Computing and the Henry Ellis Warren Professor of Electrical Engineering and Computer Science.

The building will also accommodate 50 computing research groups, which correspond to the number of new faculty the college is hiring — 25 in core computing positions and 25 in shared positions with departments at MIT.

Organized by various MIT faculty, the 12 sessions in the series delved into exciting areas of com…

In biomedicine, segmentation involves annotating pixels from an important structure in a medical image, like an organ or cell.

However, these models typically only provide one answer, while the problem of medical image segmentation is often far from black and white.

Rakic is lead author of a paper with others at MIT, the Broad Institute of MIT and Harvard, and Massachusetts General Hospital that introduces a new AI tool that can capture the uncertainty in a medical image.

Addressing ambiguityAI systems for medical image segmentation typically use neural networks.

The researchers also developed a version of Tyche that can be used with an existing, pretrained model for medical image segmentat…

To prevent this and other safety issues, companies that build large language models typically safeguard them using a process called red-teaming.

But this only works effectively if engineers know which toxic prompts to use.

The technique outperformed human testers and other machine-learning approaches by generating more distinct prompts that elicited increasingly toxic responses.

This trial-and-error process rewards the red-team model for generating prompts that trigger toxic responses from the chatbot being tested.

“If you are releasing a new AI model and are concerned about whether it will behave as expected, consider using curiosity-driven red-teaming,” says Agrawal.

Geologic hydrogen, as it’s known, is produced when water reacts with iron-rich rocks, causing the iron to oxidize.

Abate is looking to jump-start the natural hydrogen production process, implementing “proactive” approaches that involve stimulating production and harvesting the gas.

In December, French President Emmanuel Macron said his government would provide funding to explore natural hydrogen.

The projects themselves are diverse, ranging from applying industrial oil and gas methods for hydrogen production and extraction to developing models to understand hydrogen formation in rocks.

The lab-scale device will also inform the design of a real-world reactor that can accelerate hydrogen prod…

Modeling Extremely Large Images with xTAs computer vision researchers, we believe that every pixel can tell a story.

However, there seems to be a writer’s block settling into the field when it comes to dealing with large images.

Today, we make one of two sub-optimal choices when handling large images: down-sampling or cropping.

Why bother handling large images anyways?

That’s basically what we do with large images with $x$T.

As computer vision researchers, we believe that every pixel can tell a story. However, there seems to be a writer’s block settling into the field when it comes to dealing with large images. Large images are no longer rare—the cameras we carry in our pockets and those orbiting our planet snap pictures so big and detailed that they stretch our current best models and hardware to their breaking points when handling them. Generally, we face a quadratic increase in memory usage as a function of image size.

Today, we make one of two sub-optimal choices when handling large images: down-sampling or cropping. These two methods incur significant losses in the amount of information and context present…

Every year, the Berkeley Artificial Intelligence Research (BAIR) Lab graduates some of the most talented and innovative minds in artificial intelligence and machine learning. Our Ph.D. graduates have each expanded the frontiers of AI research and are now ready to embark on new adventures in academia, industry, and beyond.

These fantastic individuals bring with them a wealth of knowledge, fresh ideas, and a drive to continue contributing to the advancement of AI. Their work at BAIR, ranging from deep learning, robotics, and natural language processing to computer vision, security, and much more, has contributed significantly to their fields and has had transformative impacts on society.

This…

2024 BAIR Graduate DirectoryEvery year, the Berkeley Artificial Intelligence Research (BAIR) Lab graduates some of the most talented and innovative minds in artificial intelligence and machine learning.

Our Ph.D. graduates have each expanded the frontiers of AI research and are now ready to embark on new adventures in academia, industry, and beyond.

These fantastic individuals bring with them a wealth of knowledge, fresh ideas, and a drive to continue contributing to the advancement of AI.

Join us in celebrating the achievements of BAIR’s latest PhD graduates.

Thank you to our friends at the Stanford AI Lab for this idea!

In this post, we analyze the trend toward compound AI systems and what it means for AI developers.

We argue that compound AI systems will likely be the best way to maximize AI results in the future, and might be one of the most impactful trends in AI in 2024.

We define a Compound AI System as a system that tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools.

Developing Compound AI SystemsWhile compound AI systems can offer clear benefits, the art of designing, optimizing, and operating them is still emerging.

However, new compound AI systems contain non-differentiable components like search engines or code interpreters, a…

AI caught everyone’s attention in 2023 with Large Language Models (LLMs) that can be instructed to perform general tasks, such as translation or coding, just by prompting. This naturally led to an intense focus on models as the primary ingredient in AI application development, with everyone wondering what capabilities new LLMs will bring.

As more developers begin to build using LLMs, however, we believe that this focus is rapidly changing: state-of-the-art AI results are increasingly obtained by compound systems with multiple components, not just monolithic models.

For example, Google’s AlphaCode 2 set state-of-the-art results in programming through a carefully engineered system that uses L…

Ghostbuster: Detecting Text Ghostwritten by Large Language ModelsThe structure of Ghostbuster, our new state-of-the-art method for detecting AI-generated text.

Large language models like ChatGPT write impressively well—so well, in fact, that they’ve become a problem.

Existing tools to detect AI-generated text sometimes do poorly on data that differs from what they were trained on.

Our recent paper introduces Ghostbuster, a state-of-the-art method for detecting AI-generated text.

Many current AI-generated text detection systems are brittle to classifying different types of text (e.g., different writing styles, or different text generation models or prompts).

The structure of Ghostbuster, our new state-of-the-art method for detecting AI-generated text. Large language models like ChatGPT write impressively well—so well, in fact, that they’ve become a problem. Students have begun using these models to ghostwrite assignments, leading some schools to ban ChatGPT. In addition, these models are also prone to producing text with factual errors, so wary readers may want to know if generative AI tools have been used to ghostwrite news articles or other sources before trusting them.

What can teachers and consumers do? Existing tools to detect AI-generated text sometimes do poorly on data that differs from what they were trained on. In addition, if these m…

Asymmetric Certified Robustness via Feature-Convex Neural Networks TLDR: We propose the asymmetric certified robustness problem, which requires certified robustness for only one class and reflects real-world adversarial scenarios. This focused setting allows us to introduce feature-convex classifiers, which produce closed-form and deterministic certified radii on the order of milliseconds. Figure 1. Illustration of feature-convex classifiers and their certification for sensitive-class inputs. This architecture composes a Lipschitz-continuous feature map $\varphi$ with a learned convex function $g$. Since $g$ is convex, it is globally underapproximated by its tangent plane at $\varphi(x)$, y…

Asymmetric Certified Robustness via Feature-Convex Neural NetworksAsymmetric Certified Robustness via Feature-Convex Neural NetworksTLDR: We propose the asymmetric certified robustness problem, which requires certified robustness for only one class and reflects real-world adversarial scenarios.

We argue that these issues can be addressed by refining the certified robustness problem to be more aligned with practical adversarial settings.

The Asymmetric Certified Robustness ProblemCurrent certifiably robust classifiers produce certificates for inputs belonging to any class.

Feature-convex classifiersWe propose feature-convex neural networks to address the asymmetric robustness problem.

Conclu…

Goal Representations for Instruction FollowingGoal Representations for Instruction FollowingA longstanding goal of the field of robot learning has been to create generalist agents that can perform tasks for humans.

Goal Representations for Instruction FollowingThe GRIF model consists of a language encoder, a goal encoder, and a policy network.

Our approach, Goal Representations for Instruction Following (GRIF), jointly trains a language- and a goal- conditioned policy with aligned task representations.

In particular, we exploit this structure by requiring that language- and goal- representations be similar for the same semantic task.

We train dual image and text encoders by doing contrastiv…

Goal Representations for Instruction Following A longstanding goal of the field of robot learning has been to create generalist agents that can perform tasks for humans. Natural language has the potential to be an easy-to-use interface for humans to specify arbitrary tasks, but it is difficult to train robots to follow language instructions. Approaches like language-conditioned behavioral cloning (LCBC) train policies to directly imitate expert actions conditioned on language, but require humans to annotate all training trajectories and generalize poorly across scenes and behaviors. Meanwhile, recent goal-conditioned approaches perform much better at general manipulation tasks, but do not e…

Rethinking the Role of PPO in RLHF TL;DR: In RLHF, there’s tension between the reward learning phase, which uses human preference in the form of comparisons, and the RL fine-tuning phase, which optimizes a single, non-comparative reward. What if we performed RL in a comparative way? Figure 1: This diagram illustrates the difference between reinforcement learning from absolute feedback and relative feedback. By incorporating a new component - pairwise policy gradient, we can unify the reward modeling stage and RL stage, enabling direct updates based on pairwise responses. Large Language Models (LLMs) have powered increasingly capable virtual assistants, such as GPT-4, Claude-2, Bard and Bing…

Rethinking the Role of PPO in RLHFRethinking the Role of PPO in RLHFTL;DR: In RLHF, there’s tension between the reward learning phase, which uses human preference in the form of comparisons, and the RL fine-tuning phase, which optimizes a single, non-comparative reward.

By incorporating a new component - pairwise policy gradient, we can unify the reward modeling stage and RL stage, enabling direct updates based on pairwise responses.

Proximal Policy Optimization (PPO), the dominant RL optimizer in this process, has been reported to exhibit instability and implementation complications.

Derivation of P3OOur idea stems from the vanilla policy gradient (VPG).

Under this framework, we develop …

Goal Representations for Instruction FollowingGoal Representations for Instruction FollowingA longstanding goal of the field of robot learning has been to create generalist agents that can perform tasks for humans.

Goal Representations for Instruction FollowingThe GRIF model consists of a language encoder, a goal encoder, and a policy network.

Our approach, Goal Representations for Instruction Following (GRIF), jointly trains a language- and a goal- conditioned policy with aligned task representations.

In particular, we exploit this structure by requiring that language- and goal- representations be similar for the same semantic task.

We train dual image and text encoders by doing contrastiv…

This post discusses techniques to align them with company values and build a custom reward model using Amazon SageMaker.

There is a general perception that reward models are often associated only with Reinforcement Learning from Human Feedback (RLHF).

Delete the deployed SageMaker models, if any, and stop the SageMaker Studio notebook you launched for this exercise.

After the reward model is trained, you can use the reward model to evaluate the LLM’s responses against your subjective organizational standards.

Organizations need flexible ML pipelines that continually retrain reward models with updated rewards reflecting latest priorities and needs.

Amazon Personalize makes it straightforward to personalize your website, app, emails, and more, using the same machine learning (ML) technology used by Amazon, without requiring ML expertise.

Return item metadata in inference responses – The new recipes enable item metadata by default without extra charge, allowing you to return metadata such as genres, descriptions, and availability in inference responses.

ConclusionThe new Amazon Personalize User-Personalization-v2 and Personalized-Ranking-v2 recipes take personalization to the next level with support of larger item catalogs, reduced latency, and optimized performance.

For more information about Amazon Personalize, see the Amazon Personal…

Today, we are happy to announce Amazon Titan Text Embeddings V2, our second-generation embeddings model for Amazon Bedrock.

Benefits of Amazon Titan Text Embeddings V2Amazon Titan Text Embeddings V2 is the second-generation embedding model for Amazon Bedrock, optimized for some of the most common customer use cases we have seen with our customers.

The Massive Text Embedding Benchmark (MTEB) evaluates text embedding models across a wide range of tasks and datasets.

Benchmark resultsThe Amazon Titan Text Embeddings V2 model has the ability to output embeddings of various size.

Use Amazon Titan Text Embeddings V2 on Amazon BedrockThe new Amazon Titan Text Embeddings V2 model is available throu…

To learn more about Llama 2 on AWS, refer to Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart.

For more information on Trainium Accelerator chips, refer to Achieve high performance with lowest cost for generative AI inference using AWS Inferentia2 and AWS Trainium on Amazon SageMaker.

This allows you to capture the loss of the training job to determine when the training job should be stopped to identify the convergence of the model for optimal training.

This allows you to capture the loss of the training job to determine when the training job should be stopped to identify the convergence of the model for optimal training.

To learn more about the resiliency…

In this post, we show how Amazon Bedrock and Amazon SageMaker Canvas, a no-code AI suite, allow business users without deep technical expertise to fine-tune and deploy LLMs.

You can transform customer interaction using datasets like product Q&As with just a few clicks using Amazon Bedrock and Amazon SageMaker JumpStart models.

PrerequisitesFirst-time users need an AWS account and AWS Identity and Access Management (IAM) role with SageMaker, Amazon Bedrock, and Amazon Simple Storage Service (Amazon S3) access.

Complete the following steps to deploy your model:On the Amazon Bedrock console, choose Foundation models in the navigation pane, then choose Custom models.

For further inspiration, se…

We developed the Document Translation app, which uses Amazon Translate, to address these issues.

It recruited local bilingual volunteers to assess the quality of the machine translations against their first languages, and Amazon Translate came out on top.”The Document Translation app uses Amazon Translate for performing translations.

Amazon Translate provides high-quality document translations for contextual, accurate, and fluent translations.

Simply Readable enables you to create Easy Read documents with generative artificial intelligence (AI) using Amazon Bedrock.

Business leaders should evaluate solutions like Amazon Translate to overcome language barriers and share their brand.

Agents for Amazon Bedrock is a generative AI tool offered through Amazon Bedrock that enables generative AI applications to execute multistep tasks across company systems and data sources.

This post demonstrates how to build a chatbot using Amazon Bedrock including Agents for Amazon Bedrock and Knowledge Bases for Amazon Bedrock, within an automated solution.

In this section, we discuss some of the key Amazon Bedrock features and resources that we use in our solution.

Agents for Amazon BedrockAgents for Amazon Bedrock enables you to build and configure autonomous agents in your application.

Each response from an Amazon Bedrock agent is accompanied by a trace that details the steps being orc…

IAM Identity Center stores the user identity, is the authoritative source of identity information for Amazon Q Business applications, and validates the user’s identity when they access an Amazon Q Business application.

IAM Identity Center instanceAn Amazon Q Business application requires an IAM Identity Center instance to be associated with it.

Configure an Amazon Q Business application with IAM Identity Center enabledComplete the following steps to create an Amazon Q Business application and enable IAM Identity Center:On the Amazon Q Business console, choose Create application.

For more information about Amazon Q Business retrievers, refer to Creating and selecting a retriever for an Amazo…

To address these issues, we launched a generative artificial intelligence (AI) call summarization feature in Amazon Transcribe Call Analytics.

You can also use generative call summarization through Amazon Transcribe Post Call Analytics Solution for post-call summaries.

PrerequisitesTo get started, upload your recorded file or the sample file provided to an Amazon Simple Storage Service (Amazon S3) bucket.

Create a new Post call analytics jobComplete the following steps to create a new Post call analytics job:On the Amazon Transcribe console, choose Post-call Analytics in the navigation pane under Amazon Transcribe Call Analytics.

Generative call summarization in Amazon Transcribe Call Analy…

I’m excited to share that Amazon Q Developer, Amazon Q Business, and Amazon Q in QuickSight are available today along with several new features.

One of the most exciting announcements today is a feature of Amazon Q Business called Amazon Q Apps.

Creating an application with Amazon Q Apps is straightforward—employees can describe the type of app they want in natural language, or just tell Amazon Q Apps to do it from a conversation where Amazon Q helped solve a problem.

The Amazon Q Business Pro subscription at $20 per user/per month provides users access to the full suite of Amazon Q Business capabilities, including access to Amazon Q Apps and Amazon Q in QuickSight (Reader Pro).

We’re excit…

We built Amazon Q Business and Amazon Q in QuickSight to make this much simpler.

Watch use cases of Amazon Q Business through its simple web base interface.

If you want to learn more about how to set up and administer Q Business, check out the News Blog: Amazon Q Business.

We’re really excited to share Amazon Q Business and Amazon Q in QuickSight with you.

About the AuthorsMukesh Karki is GM of Amazon Q Business.

First, the AWS Trainium accelerator provides a high-performance, cost-effective, and readily available solution for training and fine-tuning large models.

How Metaflow integrates with TrainiumFrom a Metaflow developer perspective, using Trainium is similar to other accelerators.

Solution overviewAs a practical example, let’s set up the complete stack required to pre-train Llama2 for a few epochs on Trainium using Metaflow.

DeploymentTo deploy a Metaflow stack using AWS CloudFormation, complete the following steps:Download the CloudFormation template.

Based in Canada, he helps customers deploy and optimize deep learning training and inference workloads using AWS Inferentia and AWS Trainium.

Today, we are excited to announce that Cohere Command R and R+ foundation models are available through Amazon SageMaker JumpStart to deploy and run inference.

What are Cohere Command R and Command R+?

Command R family – include Command R and Command R+ models – are optimized for RAG based workflows such as conversational interaction and long context tasks, enabling companies to move beyond proof of concept and into production.

The Cohere Command R and R+ models are available in the Cohere hub.

To find the Command R and R+ models, search for “Command R” in the search box located at the top left of the SageMaker JumpStart landing page.

In this post, we show you how efficient we make our continual pre-training by using Trainium chips.

These projects showcase the power of domain adaptation pre-training in enhancing model performance across diverse fields, from medical applications to industrial chip design.

Join us as we unveil the future of domain-specific model adaptation and the potential of CPT with Trainium in optimizing model performance for real-world applications.

Now you can launch a training job to submit a model training script as a slurm job.

These technologies not only facilitate more effective model training, but also open up new avenues for innovation and practical application in AI-driven industries.

NVIDIA NIM enables fast and easy generative AI development and deployment for the thousands of healthcare and life sciences companies using AWS.

Harnessing optimized AI models for healthcare is easier than ever as NVIDIA NIM, a collection of cloud-native microservices, integrates with Amazon Web Services.

NIM, part of the NVIDIA AI Enterprise software platform available on AWS Marketplace, enables developers to access a growing library of AI models through industry-standard application programming interfaces, or APIs.

The company, which uses NVIDIA GPUs through AWS, is adopting NVIDIA NIM and NVIDIA ACE microservices to power a generative AI agent for digital health.

Experiment with NVIDIA …

Plus, game across more devices than ever as GeForce NOW adds improved support on Steam Deck this GFN Thursday.

Members who want to stream their favorite PC games to Valve’s Steam Deck now have an easier way to get started.

The latest GeForce NOW update, released last week, also allows members to navigate GeForce NOW on a browser using a gamepad, including on the Steam Deck.

Steam Deck is just one of many popular handheld PC devices with support for GeForce NOW.

Check out the list of new games this week:Hellblade: Senua’s Sacrifice (Steam and Xbox, available on PC Game Pass)(Steam and Xbox, available on PC Game Pass) Stormgate Closed Beta (New release on Steam, April 30, sign up for access)(…

Arthur.ai enhances the performance of AI systems across various metrics like accuracy, explainability and fairness.

In this episode of the NVIDIA AI Podcast, recorded live at GTC 2024, host Noah Kravitz sits down with Adam Wenchel, cofounder and CEO of Arthur, to discuss the challenges and opportunities of deploying generative AI.

Their conversation spans a range of topics, including AI bias, the observability of AI systems and the practical implications of AI in business.

Subscribe to the AI PodcastGet the AI Podcast through iTunes, Google Podcasts, Google Play, Amazon Music, Castbox, DoggCatcher, Overcast, PlayerFM, Pocket Casts, Podbay, PodBean, PodCruncher, PodKicker, Soundcloud, Spotif…

His recent Interactive Animatronic GLaDOS project was among nine winners in the Hackster AI Innovation Challenge.

Other top winners included contestants Andrei Ciobanu and Allen Tao, who took first prize in the generative AI models for the edge and AI at the edge applications categories, respectively.

Ciobanu used generative AI to help virtually try on clothes, while Tao developed a ROS-based robot to map the inside of a home to help find things.

He built his interactive GLaDOS robot to create a personal assistant for himself in the lab.

He used AI models such as YOLOv5, SAM and OpenPose to extract and refine data from images and videos.

Chatbots powered by large-language AI models have transformed computing, and NVIDIA ChatRTX lets users interact with their local data, accelerated by NVIDIA RTX-powered Windows PCs and workstations.

The NVIDIA RTX Remix beta update brings NVIDIA DLSS 3.5 with Ray Reconstruction to the modding platform for even more impressive real-time ray tracing.

Say It Out LoudChatRTX uses retrieval-augmented generation, NVIDIA TensorRT-LLM software and NVIDIA RTX acceleration to bring chatbot capabilities to RTX-powered Windows PCs and workstations.

With CLIP support in ChatRTX, users can interact with photos and images on their local devices through words, terms and phrases, without the need for comple…

At GTC 2024, experts from NVIDIA and our partners shared insights about GPU-accelerated tools, optimizations, and best practices for data scientists.

From the hundreds of sessions covering various topics, we’ve handpicked the top three data science sessions that you won’t want to miss.

The team also presented exciting updates on how you can tap into accelerated computing, as well as the RAPIDS roadmap for 2024.

Insights from Kaggle Grandmasters and Experts on Competitive AI and LLM FrontiersSpeakers: Chris Deotte, Senior Data Scientist at NVIDIA; David Austin, Principal Data Scientist at NVIDIA; Kazuki Onodera, Senior Deep Learning Data Scientist at NVIDIA; Jiwei Liu, Senior Data Scientist …

Linz, Austria-based startup, is revolutionizing maritime safety with NVIDIA's edge AI and computer vision technology, creating a safer journey on every wave.

Thanks to SEA.AI, commercial and recreational oceangoers worldwide can travel with more confidence.

When combined with high-tech optical sensors and the latest AI technology from NVIDIA, SEA.AI’s systems can recognize and classify objects in real-time, significantly improving maritime safety.

SEA.AI technology can detect a person in water up to 700 meters — almost half a mile — away, a dinghy up to 3,000 meters, and motorboats up to 7,500 meters.

One, SEA.AI Sentry, provides 360-degree situational awareness for commercial vessels and m…

All 2024 models are equipped with four NVIDIA DRIVE Orin systems-on-a-chip (SoCs).

The largest EV maker in the world, BYD is building both its Ocean and Dynasty series on NVIDIA DRIVE Orin.

The entire lineup of the IM L6 is equipped with NVIDIA DRIVE Orin to power intelligent driving abilities, including urban NOA features.

Momenta is rolling out a new NVIDIA DRIVE Orin solution to accelerate commercialization of urban NOA capabilities at scale.

WeRide is exhibiting Chery’s Exeed Sterra ET SUV and ES sedan, both powered by NVIDIA DRIVE Orin.

‘StarCraft Remastered,’ ‘StarCraft II,’ ‘Diablo II: Resurrected’ and ‘Diablo III’ lead 16 new games now available from the cloud.

Support for Battle.net on GeForce NOW expands this GFN Thursday, as titles from the iconic StarCraft and Diablo series come to the cloud.

StarCraft Remastered, StarCraft II, Diablo II: Resurrected and Diablo III are part of 16 new games joining the GeForce NOW library of more than 1,900 titles.

StarCraft Remastered, StarCraft II, Diablo II: Resurrected and Diablo III bring galactic warfare, epic quests and legendary battles to the cloud.

These top games join the Battle.net games first added to GeForce NOW, including Diablo IV, Overwatch 2, Call of Duty HQ and Hea…

Into the Omniverse: Unlocking the Future of Manufacturing With OpenUSD on Siemens Teamcenter XEditor’s note: This post is part of Into the Omniverse, a series focused on how artists, developers and enterprises can transform their workflows using the latest advances in OpenUSD and NVIDIA Omniverse.

Siemens is adding support for OpenUSD in its Siemens Xcelerator platform applications, starting with Teamcenter X software.

Ian Fisher, a member of Siemens Digital Industries Software team, is no stranger to the impact of embracing digital transformation — especially one powered by OpenUSD and generative AI.

Siemens’ adoption of OpenUSD means that companies like HD Hyundai, a leader in sustainable…

Manufacturers are using virtual factories to unlock new possibilities from planning to operations.

A virtual factory is a physically accurate representation of a real factory.

: During facility design, construction and commissioning, virtual factories allow project stakeholders to visualize designs in the context of the entire facility and production process.

Intelligent and Optimized Operations: Operations teams can integrate their virtual factories with valuable production data from Internet of Things technology at the edge, and tap AI to drive further optimizations.

Virtual Factories: A Testing Ground for AI and RoboticsRobotics developers are increasingly using virtual factories to trai…

Run:ai customers include some of the world’s largest enterprises across multiple industries, which use the Run:ai platform to manage data-center-scale GPU clusters.

NVIDIA DGX and DGX Cloud customers will gain access to Run:ai’s capabilities for their AI workloads, particularly for large language model deployments.

Run:ai’s solutions are already integrated with NVIDIA DGX, NVIDIA DGX SuperPOD, NVIDIA Base Command, NGC containers, and NVIDIA AI Enterprise software, among other products.

Together with Run:ai, NVIDIA will enable customers to have a single fabric that accesses GPU solutions anywhere.

Customers can expect to benefit from better GPU utilization, improved management of GPU infrast…

Through ongoing research and collaboration, Bretherton and his team aim to improve climate modeling and enable society to better mitigate and adapt to the impacts of climate change.

Stay tuned for more episodes recorded live from GTC, and watch the replay of Bretherton’s GTC session on using machine learning for climate modeling.

Time Stamps2:03: What is climate modeling and how can it prepare us for climate change?

5:28: How can machine learning help enhance climate modeling?

25:59: How do you measure the accuracy or performance of an emulator that’s doing something like climate modeling out into the future?

DLSS 3.5 with Ray Reconstruction creates higher quality ray-traced images for intensive ray-traced games and apps.

DLSS 3.5 Ray Reconstruction introduces an NVIDIA supercomputer-trained, AI-powered neural network that generates higher-quality pixels in between the sampled rays.

Deep Learning, Deep GamingRay Reconstruction is just one of the AI graphics breakthroughs that multiply performance in DLSS.

Combining the DLSS-generated frames with DLSS Super Resolution enables DLSS 3 to reconstruct seven-eighths of the displayed pixels with AI, boosting frame rates by up to 4x compared to without DLSS.

Because DLSS Frame Generation is post-processed (applied after the main render) on the GPU, it c…

NVIDIA DGX CloudTo help address the need for AI training, NVIDIA DGX Cloud offers state-of-the-art accelerated computing resources for users to access without the hassle of sourcing, setting up, and configuring the infrastructure themselves.

In this post, I’m excited to introduce Union’s NVIDIA DGX Agent, which enables you to integrate your Flyte workflows with NVIDIA DGX Cloud.

In the context of AI model development, workflows have become the de facto abstraction for managing the complexity of data and model management.

Fine-tuning a pipeline using FlyteIntroducing Union’s NVIDIA DGX AgentLet’s say you’re working on fine-tuning the Mixtral 8x7b model.

To leverage DGX Cloud to get around th…

To help personalize content, tailor and measure ads and provide a safer experience, we use cookies.

By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies.

Learn more, including about available controls: Cookie PolicyAccept

The next generation of Meta’s large-scale infrastructure is being built with AI in mind, including supporting new generative AI (GenAI) products and services, recommendation systems, and advanced AI research.

It’s an investment we expect will grow in the years ahead as the compute requirements to support AI models increase alongside the models’ sophistication.

Last year, we unveiled the Meta Training and Inference Accelerator (MTIA) v1, our first-generation AI inference accelerator that we designed in-house with Meta’s AI workloads in mind – specifically our deep learning recommendation models that are improving a variety of experiences across our products.

MTIA is a long-term venture to pr…

Bandwidth estimation (BWE) and congestion control play an important role in delivering high-quality real-time communication (RTC) across Meta’s family of apps.

Network characterizationAn ML model-based approach leverages time series data to improve the bandwidth estimation by using offline parameter tuning for characterized network types.

The first component is offline ML model learning using ML to categorize the network type (random packet loss versus bursty loss).

The non-time series data or dense data will pass through a dense layer (i.e., a fully connected layer).

Use case: Random packet loss classificationLet’s consider the use case of categorizing packet loss as either random or conge…

In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases.

AI training debugging with LogarithmBefore looking at Logarithm’s internals, we present support for training systems and model issue debugging, one of the prominent use cases of Logarithm at Meta.

ML model training workflows tend to have a wide range of failure modes, spanning data inputs, model code and hyperparameters, and systems components (e.g., PyTorch, data readers, checkpointing, framework code, and hardware).

Logarithm ingests both systems logs from the training stack, and model telemetry from training jobs that the stack executes.

Filter–by-callsite enables hiding known lo…

While we’ve had a long history of building AI infrastructure, we first shared details on our AI Research SuperCluster (RSC), featuring 16,000 NVIDIA A100 GPUs, in 2022.

Under the hoodOur newer AI clusters build upon the successes and lessons learned from RSC.

Our out-of-box performance for large clusters was initially poor and inconsistent, compared to optimized small cluster performance.

Commitment to open AI innovationMeta maintains its commitment to open innovation in AI software and hardware.

The future of Meta’s AI infrastructureThese two AI training cluster designs are a part of our larger roadmap for the future of AI.

These improvements helped us find and remove many unnecessary dependencies, making build graph analysis and overall build times much better.

In response to this challenge, we implemented a new solution for the packaging and distribution of Python executables – the Content Addressable Filesystem (CAF).

LazyCAF and enforcing uniform revisions: Areas for further ML iteration improvementsThe improvements we implemented have proven highly effective, drastically reducing the overhead and significantly elevating the efficiency of our ML engineers.

Faster build times and more efficient packaging and distribution of executables have reduced overhead by double-digit percentages.

We plan to enable all…

At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime.

The challenges of adopting Lazy ImportsWhile Lazy Imports’ approach significantly improved ML development, it was not all a bed of roses.

With Lazy Imports, Meta’s ML developers are now equipped to work more efficiently, experiment more rapidly, and achieve results faster.

Here’s a glimpse into our future endeavors:Streamlining developer onboardingThe learning curve associated with Lazy Imports can be a challenge for newcomers.

Building a robust community that helps supporting paradigms and patterns that play well with Lazy Imports is one of our future …

What’s going on with generative AI (GenAI) at Meta?

In this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) speaks with Devi Parikh, an AI research director at Meta.

They cover a wide range of topics, including the history and future of GenAI and the most interesting research papers that have come out recently.

And, of course, they discuss some of Meta’s latest GenAI innovations, including:Audiobox, a foundational model for generating sound and soundscapes using natural language prompts.

And if you’re interested in AI career opportunities at Meta visit the Meta Careers page.

In this post, we will provide an overview of the end-to-end debugging workflows supported by HawkEye, components of the system, and the product surface for Meta product and monetization teams to debug AI model and feature issues.

HawkEye includes infrastructure for continuously collecting data on serving and training models, data generation, and analysis components for mining root causes.

However, significant differences indicate problems with either the training data or loss divergence (e.g., loss or gradient explosion) in the bad snapshot.

Such issues can happen for several hard-to-diagnose reasons, ranging from the complex data pipelines behind training data, to data corruptions.

HawkEye…

The 2023 edition of Networking at Scale focused on how Meta’s engineers and researchers have been designing and operating the network infrastructure over the last several years for Meta’s AI workloads, including our numerous ranking and recommendation workloads and the immense Generative AI models.

But the sheer scale and complexity of GenAi models means new challenges for Meta’s network infrastructure.

Meta’s Network Journey to Enable AIHany Morsy, Network EngineerSusana Contrera, Network EngineerOver the years, Meta’s AI infrastructure has transitioned from CPU-based to GPU-based training due to growing AI workloads.

Traffic Engineering for AI Training NetworksShuqiang Zhang, Software Eng…

Fueling the success of these products are world-class infrastructure teams, including Meta’s custom AI silicon team, led by Olivia Wu, a leader in the silicon industry for 30 years.

In 2018, I saw a social media post from Yann LeCun, our Chief AI Scientist, that Meta was looking for someone to help build AI silicon in-house.

I knew of just a few other companies designing their own custom AI silicon, but they were mainly focused only on silicon and not the software ecosystem and products.

What’s next for the AI silicon design team?

We’re continuing to gather feedback and input from our AI software teams to shape the features of our future AI silicon.

Meta presents Chakra execution traces , an open graph-based representation of AI/ML workload execution, laying the foundation for benchmarking and network performance optimization.

The limitations of traditional AI benchmarking methodologyTraditionally, benchmarking AI systems has largely relied on running full ML workloads.

How Meta leverages Chakra execution tracesAt Meta, we collect execution traces from our production servers every day.

Future plansEnhancing the benchmarking capability of Chakra execution tracesWhile the execution trace replayer enables replay of execution traces, it brings forth challenges.

Using AI to generate representative execution tracesChakra execution traces are…

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters.

Arcadia gives Meta’s researchers and engineers valuable insights into the performance of AI models and workloads in an AI cluster – enabling data-driven decision making in the design of AI clusters.

That’s where Arcadia, Meta’s end-to-end AI system performance simulator, comes in.

By providing insights into the impact of these factors on system performance, Arcadia facilitates data-driven decision-making processes and fosters the evolution of models and hardware.

This comprehensive set of metrics empowers stakeholders to analyze the impact of different factor…

Today, we are releasing Code Llama, a large language model (LLM) that can use text prompts to generate code.

Additionally, we have further fine-tuned two additional variations of Code Llama: Code Llama - Python and Code Llama - Instruct.

Code Llama - Python is a language-specialized variation of Code Llama, further fine-tuned on 100B tokens of Python code.

Code Llama - Instruct is an instruction fine-tuned and aligned variation of Code Llama.

We recommend using Code Llama - Instruct variants whenever using Code Llama for code generation since Code Llama - Instruct has been fine-tuned to generate helpful and safe answers in natural language.

[...]

Read More...

The post Meta Connect 2023: September 27 – 28 appeared first on Engineering at Meta.

We can further control the output of LLMs through parameters such as “temperature” or a “frequency penalty,” which influence an LLM’s output on a token-by-token basis.

How the Greedy Decoding, Beam Search, and Sampling post-processing techniques determine the next token to output.

How advanced techniques like frequency penalties, logit bias, and structured output give you even more control over an LLM’s output.

Before we dive into post-processing techniques for customizing LLM outputs, it’s crucial to understand how an LLM generates its output in the first place.

Adjusting the temperature parameter modifies the softmax function, influencing the diversity and predictability of a large langua…

In this article, we’ll survey the most commonly used deep learning optimization algorithms, including Gradient Descent, Stochastic Gradient Descent, and the Adam optimizer.

Understanding different optimization algorithms and their strengths and weaknesses is crucial for any data scientist training deep learning models.

Optimization in deep learning Have a look at other articles on our blog exploring aspects of optimization in deep learning: Deep Learning Model Optimization Methods: Deep learning models exhibit excellent performance but require high computational resources.

:Mini-batch Gradient DescentMini-batch Gradient Descent strikes a balance between the thorough, calculated approach of …

On top of that, neptune.ai integrates with any MLOps stack, and it just works.

Now, with less boilerplate code, you can log and visualize information from your ZenML pipeline steps (e.g., models, parameters, metrics).

You’re looking for a more visually interactive way of navigating the results produced from your ZenML pipeline runs (e.g., models, metrics, datasets).

In this example, we log a simple ZenML pipeline to Neptune using the Experiment Tracker stack component.

The example assumes that you have ZenML installed together with the Neptune integration.

Scatter plotsIf you have two metrics or parameters that you wish to compare or see how they relate to each other throughout the runs, you can now create a scatter plot.

See an example of a scatter plot in the Neptune app.

Distraction-free modeYou can now view run dashboards and compare views in distraction-free mode.

(neptune 1.5.0)We added a new environment variable NEPTUNE_DATA_DIRECTORY, which lets you specify where the .neptune folder should be created instead of the current working directory.

(neptune 1.6.0)Web applicationWe fixed an issue where a field named “type” could not be displayed as a runs table column.

We are excited to announce that Neptune and Modelbit have partnered to release an integration to enable better ML model deployment and experiment tracking.

Data scientists and machine learning engineers can use the integration to train and deploy machine learning models in Modelbit while logging and visualizing training progress in Neptune.

We’ll log the model’s hyperparameters and accuracy to Neptune and then deploy the model to a REST endpoint.

First, import “modelbit” and “neptune” and authenticate your notebook with Modelbit:import modelbit, neptune mb = modelbit.login()If your “NEPTUNE_API_TOKEN” isn’t already in your notebook’s environment, add it:import os os.environ["NEPTUNE_API_TOK…

(neptune 1.9.0): When fetching runs with , you have 4 new parameters: , , , and .

(neptune 1.9.0) All messages that Neptune prints to the console are prefixed with [neptune].

(neptune 1.9.0)that Neptune prints to the console are prefixed with [neptune].

(neptune 1.9.0) We introduced querying functionality to table fetching methods.

(neptune 1.10.0)IntegrationsWe introduced the log_model_summary parameter to NeptuneCallback() ( neptune- tensorflow-keras 2.2.1)parameter to ( neptune- 2.2.1) We added support for logging project requirements with the dependencies parameter.

This page lists all functions, parameters, and constants exposed by the Neptune Python API.

(kedro-neptune 0.3.0)Web applicationWhen modifying a search query in the runs table, you can now edit the operator and value without changing the other parts.

You can now click on a username in the Owner column (in the runs table) to instantly create a filter for runs created by that account.

column (in the runs table) to instantly create a filter for runs created by that account.

(neptune 1.8.6)Web applicationWe fixed an issue where a field named “type” could not be displayed as a runs table column.

Strategies for improving GPU usage include mixed-precision training, optimizing data transfer and processing, and appropriately dividing workloads between CPU and GPU.

The GPU memory usage metric reflects the amount of memory allocated to the GPU relative to its total memory capacity.

Optimizing data preprocessingIn addition to I/O and data transfer to the GPU, data preprocessing can become a bottleneck for GPU utilization.

This indicates that by improving GPU utilization, training could be sped up, and GPU resources could be used more efficiently.

However, across the many projects I’ve worked on, the following guidelines for optimizing GPU usage have often proven helpful:Always monitor GPU…

The role of zero-shot and few-shot learning in LLMsThe goal of zero-shot and few-shot learning is to get a machine-learning model to perform a new task it was not trained for.

“Few-shot learning” and “zero-shot learning” are well-known concepts in machine learning that were studied long before LLMs appeared on the scene.

In the context of LLMs, these terms are sometimes used interchangeably with “few-shot prompting” and “zero-shot prompting.” However, they are not the same.

Where zero-shot prompting failsLet’s turn to two use cases where zero-shot prompting is insufficient.

Where few-shot prompting failsFinally, let’s look at a situation where few-shot prompting won’t get us far.

Large Language Models (LLMs) like Meta AI’s LLaMA models, MISTRAL AI’s open models, and OpenAI’s GPT series have improved language-based AI.

Monitoring Monitor model performance for data drift and model degradation, often using automated monitoring tools.

Automate prompt selection: Implement systems that automatically choose the best prompt for a given task using historical data on prompt performance and the specifics of the current request.

Monitoring and observability are about tracking LLMs’ performance, health, and operational metrics in production to ensure they perform optimally and reliably.

Use platforms that offer comprehensive observability into LLM performance, including function…

The canonical MLflow setup for teams: The MLflow client embedded in the training code communicates with the MLflow tracking server, which handles access to cloud storage (artifact store) and a database (metadata store).

| Modified based on: sourceDeploying the MLflow tracking serverMLflow’s tracking server is relatively lightweight.

With this in mind, there are three main options for deploying the MLflow tracking server on AWS:Deploying the MLflow tracking server on an Amazon EC2 instance.

Recommended The Best MLflow alternatives Read alsoSetting up an artifact storeThe artifact store is the third relevant cost item in an MLflow deployment.

Further, everyone who has access to your MLflow tr…

Knowledge distillation transfers insights from a complex “teacher” model to a simpler “student” model, maintaining performance with less computational demand.

Distillation loss: The student model is trained not just to replicate the output of the teacher model but to match the output distributions produced by the teacher model.

Model architecture compatibility: The effectiveness of knowledge distillation depends on how well the student model can learn from the teacher model, which is greatly influenced by their architectural compatibility.

Data is fed to both a complex ‘Teacher Model’ and a simpler ‘Student Model.’ The Teacher Model, which consists of multiple layers (from Layer 1 to Layer …

Continual learning scenariosDepending on the data stream characteristics, problems within the continual learning scenario can be divided into three, each with a common solution.

Task incremental continual learningTask Incremental (TI) continual learning is classic multi-task learning but in an incremental way.

Continual learning methodsOver the second decade of the 2000s, there has been a rapid improvement in recent advances in continual learning methods.

Recommended How to Build Machine Learning Systems with a Feature Store See alsoHow to choose the right continual learning method for your projectAcross the three groups of continual learning approaches, many techniques exist.

Continual lea…

Neither teams working on proof-of-concept projects nor production ML systems are immune from job cuts.

The rapid advancements of Large Language Models (LLMs) are changing the day-to-day work of ML practitioners and how company leadership thinks about AI.

The rise of modern LLMsIn 2023, the dominance of modern LLMs became increasingly evident, challenging the incumbent classical NLP models.

Even small and relatively weaker LLMs like DistilGPT2 and t5-small have surpassed classical NLP models in understanding context and generating coherent text.

As you can see, it’s unclear whether people working on PoC or production projects are at higher risk of layoffs.

In this article, I will outline the learnings and challenges I faced while on the ML platform team at Mailchimp, building infrastructure and setting up the environment for development and testing.

Mailchimp’s ML Platform: genesis, challenges, and objectivesMailchimp is a 20-year-old bootstrapped email marketing company.

Team setup and responsibilitiesWe had around 20 data engineers and ML(Ops) engineers working on the ML platform at Mailchimp.

Recommended Learnings From Building the ML Platform at Mailchimp See full episodeIn times of generative AI, a good ML infrastructure matters a lotA lesson that I’ve learned time and time again over the past years is the enduring importance of ‘boring’…

Paper: https://arxiv.org/abs/2403.07691 Abstract:

While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empiri…

OUTLINE:

0:00 - Intro

0:19 - Our next-generation Meta Training and Inference Accelerator

01:39 - ALOHA Unleashed

03:10 - Apple Inks $50M Deal with Shutterstock for AI Training Data

04:28 - OpenAI Researchers, Including Ally of Sutskever, Fired for Alleged Leaking

05:01 - Adobe's Ethical Firefly AI was Trained on Midjourney Images

05:52 - Trudeau announces $2.4billion for AI-related investments

06:48 - RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

07:15 - CodeGemma - an official Google release for code LLMs

07:24 - Mistral AI: Cheaper, Better, Faster, Stronger

08:08 - Vezora/Mistral-22B-v0.1

09:00 - WizardLM-2, next generation state-of-the-art-LLM

09:31 - Idefic…

Paper: https://arxiv.org/abs/2404.09173 Abstract:

While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-…

OUTLINE:

0:00 - Intro

0:21 - Debunking Devin: "First AI Software Engineer" Upwork lie exposed!

07:24 - NeurIPS 2024 will have a track for papers from high schoolers.

13:29 - Opus can operate as a Turing machine.

13:47 - An AI-Powered, Self-Running Propaganda Machine for $105

14:27 - TechScape: How cheap, outsourced labour in Africa is shaping AI English

16:25 - Is ChatGPT Transforming Academics' Writing Style? References:

https://news.ycombinator.com/item?id=40008109&s=09

https://www.youtube.com/watch?v=tNmgmwEtoWE

https://www.youtube.com/watch?v=xE2fxcETP5E

https://twitter.com/itsandrewgao/status/1779369373737668669?t=omW3DvRNmZyce8oo0Ehf1g&s=09

https://twitter.com/0interestrates/status/17…

Google researchers achieve supposedly infinite context attention via compressive memory. Paper: https://arxiv.org/abs/2404.07143 Abstract:

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M …

Meta's Llama 3 is out. New model, new license, new opportunities. References:

https://llama.meta.com/llama3/

https://ai.meta.com/blog/meta-llama-3/

https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

https://llama.meta.com/trust-and-safety/

https://ai.meta.com/research/publications/cyberseceval-2-a-wide-ranging-cybersecurity-evaluation-suite-for-large-language-models/

https://github.com/meta-llama/llama-recipes/tree/main/recipes/responsible_ai

https://llama.meta.com/llama3/license/

https://about.fb.com/news/2024/04/meta-ai-assistant-built-with-llama-3/?utm_source=twitter&utm_medium=organic_social&utm_content=thread&utm_campaign=imagineflash

https://twitter.com/minchoi/status/178277…

Links:

Homepage: https://ykilcher.com

Merch: https://ykilcher.com/merch

YouTube: https://www.youtube.com/c/yannickilcher

Twitter: https://twitter.com/ykilcher

Discord: https://ykilcher.com/discord

LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):

SubscribeStar: https://www.subscribestar.com/yannickilcher

Patreon: https://www.patreon.com/yannickilcher

Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq

Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2

Litecoin (LTC): LQW2TRyKYetVC8WjFkhpP…

Some updates from industry in the Machine Learning world Links:

Homepage: https://ykilcher.com

Merch: https://ykilcher.com/merch

YouTube: https://www.youtube.com/c/yannickilcher

Twitter: https://twitter.com/ykilcher

Discord: https://ykilcher.com/discord

LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):

SubscribeStar: https://www.subscribestar.com/yannickilcher

Patreon: https://www.patreon.com/yannickilcher

Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq

Ethereum (ETH): 0x7ad3513E3B8f66799f507…

A flurry of new models continues to appear. Links:

Homepage: https://ykilcher.com

Merch: https://ykilcher.com/merch

YouTube: https://www.youtube.com/c/yannickilcher

Twitter: https://twitter.com/ykilcher

Discord: https://ykilcher.com/discord

LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):

SubscribeStar: https://www.subscribestar.com/yannickilcher

Patreon: https://www.patreon.com/yannickilcher

Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq

Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC…

Flow matching is a more general method than diffusion and serves as the basis for models like Stable Diffusion 3. Paper: https://arxiv.org/abs/2210.02747 Abstract:

We introduce a new paradigm for generative modeling built on Continuous Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale. Specifically, we present the notion of Flow Matching (FM), a simulation-free approach for training CNFs based on regressing vector fields of fixed conditional probability paths. Flow Matching is compatible with a general family of Gaussian probability paths for transforming between noise and data samples -- which subsumes existing diffusion paths as specific instances. Interestingly, …

Paper: https://arxiv.org/abs/2402.14083 Abstract:

While Transformers have enabled tremendous progress in various application settings, such architectures still lag behind traditional symbolic planners for solving complex decision making tasks. In this work, we demonstrate how to train Transformers to solve complex planning tasks and present Searchformer, a Transformer model that optimally solves previously unseen Sokoban puzzles 93.7% of the time, while using up to 26.8% fewer search steps than standard A∗ search. Searchformer is an encoder-decoder Transformer model trained to predict the search dynamics of A∗. This model is then fine-tuned via expert iterations to perform fewer search step…

OUTLINE:

0:00 - Intro

0:15 - XAI releases Grok-1

2:00 - Nvidia GTC

4:45 - Comment of the Week

5:35 - Brute-forcing OpenAI model names

7:30 - Inflection AI gets eaten by Microsoft

9:25 - EU AI Act moving forward

11:45 - Advances in Robotics

14:00 - India retracts controversial advisory

14:30 - OpenSora

15:20 - Improved Gemma fine-tuning

16:20 - Decoding encrypted LLM traffic

17:45 - Varia References:

https://x.ai/blog/grok-os

https://github.com/xai-org/grok-1

https://finance.yahoo.com/news/nvidia-debuts-next-generation-blackwell-ai-chip-at-gtc-2024-205825161.html?guccounter=1&guce_referrer=aHR0cHM6Ly9uZXdzLmdvb2dsZS5jb20v&guce_referrer_sig=AQAAAHYRVePPrDnH3HxPV8smDzUiia_ztWttteAmHKxy-x_Z75lq…

Your weekly dose of ML News OUTLINE:

0:00 - Intro

0:15 - Devin: AI software engineer

5:50 - Mira Murati on Sora training data

6:50 - Inflection accused of copying Claude

9:00 - Tools & papers

16:30 - GPT-4.5-turbo mystery

17:30 - US government report: total extinction by AI

19:20 - Various other news References:

https://www.cognition-labs.com/introducing-devin

https://twitter.com/cognition_labs/status/1767548763134964000?t=ZECIn-uqbguwHtY8X_Gvtw&s=09

https://news.google.com/stories/CAAqNggKIjBDQklTSGpvSmMzUnZjbmt0TXpZd1NoRUtEd2lWMUwyU0N4RnVWM3pSRWhWX01pZ0FQAQ?hl=en-US&gl=US&ceid=US%3Aen

https://www.bloomberg.com/news/articles/2024-03-12/cognition-ai-is-a-peter-thiel-backed-coding-assistant?…

#mlnews #ainews #openai OUTLINE:

0:00 - Intro

0:20 - Elon sues OpenAI

14:00 - Mistral Large

16:40 - ML Espionage

18:30 - More Gemini Drama

24:00 - Copilot generates spicy images

26:55 - Gemma bugs

28:45 - Varia References: https://gist.github.com/yk/0c065cdc8e414738abfaae4f8e417e00 Thumbnail pictures: Wikipedia Links:

Homepage: https://ykilcher.com

Merch: https://ykilcher.com/merch

YouTube: https://www.youtube.com/c/yannickilcher

Twitter: https://twitter.com/ykilcher

Discord: https://ykilcher.com/discord

LinkedIn: https://www.linkedin.com/in/ykilcher If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and …

Hey everyone! Thank you so much for watching this overview of Llama 3 looking at the release notes and seeing a demo of how to integrate it with DSPy through Ollama and how to use DSPy's MIPRO to find the optimal prompt when using this new large language model for RAG! We are hosting an event in San Francisco on May 1st with Arize AI and Cohere, featuring a talk from Omar Khattab, the lead author of DSPy! Hope to see you there! https://lu.ma/dspy Introducing Meta Llama 3: https://ai.meta.com/blog/meta-llama-3/ Ollama Llama 3: https://ollama.com/library/llama3 Weaviate Recipes: https://github.com/weaviate/recipes/blob/main/integrations/dspy/llms/Llama3.ipynb Chapters

0:00 Llama3!!

1:28 Relea…

Hey everyone! Thank you so much for watching this overview of Command R+ showing you how you can use the new model in DSPy and a quick RAG demo, as well as walking through the details of the release post! Congratulations to the Cohere team! Super exciting times to be working with LLM systems! Introducing Command R+: A Scalable LLM Built for Business - https://txt.cohere.com/command-r-plus-microsoft-azure/ Link to demo notebook - https://github.com/weaviate/recipes/blob/main/integrations/dspy/llms/Command-R-Plus.ipynb Chapters

0:00 Welcome! Command R+!

1:12 Demo with Cohere, DSPy, and Weaviate

6:06 Command R+ Announcement Post

9:24 LLM Evals

Unfortunately, Large Language Models will not consistently follow the instructions that you give them. This is a massive problem when you are building AI systems that require a particular type of output from the previous step to feed into the next one! For example, imagine you are building a blog post writing system that first takes a question and retrieved context to output a list of topics. These topics have to be formatted in a particular way, such as a comma-separated list or a JSON of Topic objects, such that the system can continue writing the blog post! I am SUPER excited to share the 4th video in my DSPy series, diving into 3 solutions to structuring outputs in DSPy programs: (1) **…

Hey everyone! Thank you so much for watching the 3rd edition of the DSPy series, Adding Depth to DSPy Programs!! You can find the examples and links to community resources / news on https://github.com/weaviate/recipes! Chapters

0:00 Intro

0:50 Chapters Overview

5:06 Weaviate Recipes

5:24 DSPy News and Community Notes

13:51 Adding Depth to RAG Programs

18:40 Multi-Model DSPy Programs

20:18 DSPy Optimizers

25:30 Deep Dive Optimizers

27:55 Into the Optimizer Code!

37:48 Demo #1: Adding Depth to RAG

1:05:25 Demo #2: Questions to Blogs

1:07:48 Thank you so much for watching!

Hey everyone! Thank you so much for watching this tutorial on getting started with RAG programming in DSPy! This video will take you through 4 major aspects of building DSPy programs (1) Installation, settings, and Datasets with dspy.Example, (2) LLM Metrics, (3) The DSPy programming model, and (4) Optimization!! The notebook used in the video can be found here: https://github.com/weaviate/recipes/blob/main/integrations/dspy/1.Getting-Started-with-RAG-in-DSPy.ipynb All future videos, as well as additional utils like data import scripts, will be in this folder: https://github.com/weaviate/recipes/tree/main/integrations/dspy Please leave a star, it helps a lot! DSPy on GitHub: https://github.…

Hey everyone! Thank you so much for watching this explanation of DSPy! DSPy is a super exciting new framework for developing LLM programs! Pioneered by frameworks such as LangChain and LlamaIndex, we can build much more powerful systems by chaining together LLM calls! This means that the output of one call to an LLM is the input to the next, and so on. We can think of chains as programs, with each LLM call analogous to a function that takes text as input and produces text as output. DSPy offers a new programming model, inspired by PyTorch, that gives you a massive amount of control over these LLM programs. Further the Signature abstraction wraps prompts and structured input / outputs to cle…

This comes from a full video dissecting how LLMs work. In the shorts player, you can click the link at the bottom of the screen, or for reference: https://youtu.be/wjZofJX0v4M

Demystifying attention, the key mechanism inside transformers and LLMs.

Instead of sponsored ad reads, these lessons are funded directly by viewers: https://3b1b.co/support

Special thanks to these supporters: https://www.3blue1brown.com/lessons/attention#thanks

An equally valuable form of support is to simply share the videos. Demystifying self-attention, multiple heads, and cross-attention.

Instead of sponsored ad reads, these lessons are funded directly by viewers: https://3b1b.co/support The first pass for the translated subtitles here is machine-generated, and therefore notably imperfect. To contribute edits or fixes, visit https://translate.3blue1brown.com/ ------------------ Here are …

An introduction to transformers and their prerequisites

Early view of the next chapter for patrons: https://3b1b.co/early-attention Other recommended resources on the topic. Richard Turner's introduction is one of the best starting places:

https://arxiv.org/pdf/2304.10557.pdf Coding a GPT with Andrej Karpathy

https://youtu.be/kCc8FmEb1nY Introduction to self-attention by John Hewitt

https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf History of language models by Brit Cruise:

https://youtu.be/OFS90-FX6pg ------------------ Timestamps 0:00 - Predict, sample, repeat

3:03 - Inside a transformer

6:36 - Chapter layout

7:20 - The premise of Deep Learni…

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/aXRTczANuIs

A link to the full video answering this is at the bottom of the screen. Or, for reference: https://youtu.be/LqbZpur38nw

A link to the full video answering this is at the bottom of the screen. Or, for reference: https://youtu.be/bOXCLR3Wric

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/pQa_tWZmlGs The full video this comes from proves why slicing a cone gives the same shape as the two-thumbtacks-and-string construction, which is beautiful. Editing from long-form to short by Dawid Kołodziej

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/HZGCoVF3YvM Editing from long-form to short by Dawid Kołodziej

A link to the full video about Bayesian thinking is at the bottom of the screen.

Or, for reference: https://youtu.be/lG4VkPoG3ko Long-to-short editing by Dawid Kołodziej

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/OkmNXy7er84 Editing from the original video into this short by Dawid Kołodziej

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/Cz4Q4QOuoo8 That video answers various viewer questions about the index of refraction. Editing from long-form to short by Dawid Kołodziej

A link to the full video is at the bottom of the screen. Or, for reference: https://youtu.be/ltLUadnCyi0

A link to the full video is at the bottom of the screen.

Or, for reference: https://youtu.be/KTzGBJPuJwM That video unpacks the mechanism behind how light slows down in passing through a medium, and why the slow-down rate would depend on color. Editing from long-form to short by Dawid Kołodziej

A link to the full video is at the bottom of the screen.

Or, for reference: https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab Editing from long-form to short by Dawid Kołodziej

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 The paper "Learning agile soccer skills for a bipedal robot with deep reinforcement learning" is available here:

https://sites.google.com/view/op3-soccer

https://www.science.org/doi/10.1126/scirobotics.adi8022 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Gaston Ingaramo, Gordon Child, Jo…

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers ChatGPT:

https://chat.openai.com/ Chatbot arena leaderboard:

https://chat.lmsys.org/?leaderboard Video studying Devin: https://www.youtube.com/watch?v=tNmgmwEtoWE Try this instead! https://github.com/princeton-nlp/SWE-agent 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Gaston Ingaramo, Gord…

❤️ Check out Microsoft Azure AI and try it out for free:

https://azure.microsoft.com/en-us/solutions/ai 📝 The paper "Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis " is available here:

https://research.nvidia.com/labs/nxp/lp3d/ Fully Connected conference: https://wandb.ai/site/resources/events/fully-connected MKBHD, iJustine ,Brian Tong persona source: https://www.youtube.com/watch?v=dtp6b76pMak 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous …

❤️ Check out Microsoft Azure AI and try it out for free:

https://azure.microsoft.com/en-us/solutions/ai 📝 The paper "Grandmaster-Level Chess Without Search" is available here:

https://arxiv.org/abs/2402.04494 +1:

https://www.anthropic.com/news/decomposing-language-models-into-understandable-components

https://transformer-circuits.pub/2023/monosemantic-features/index.html 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Ba…

❤️ Check out Microsoft Azure AI and try it out for free:

https://azure.microsoft.com/en-us/solutions/ai 📝 The paper "ViCMA: Visual Control of Multibody Animations" is available here:

https://research.nvidia.com/labs/prl/vicma/ 📝 The paper "Inverse-Foley Animation: Synchronizing rigid-body motions to sound" is available here:

https://www.cs.cornell.edu/projects/Sound/ifa/ 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Ba…

❤️ Check out Microsoft Azure AI and try it out for free:

https://azure.microsoft.com/en-us/solutions/ai 📝 The paper "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context" is available here:

https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf 📝 The paper "Gemma: Open Models Based on Gemini Research and Technology" is available here:

https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf Try Gemma:

https://huggingface.co/chat Sources:

https://twitter.com/skirano/status/1760468624706351383

https://twitter.com/mckaywrigley/status/1761113846520131816

https://simonwillison.net/2024/Feb/21/gemini-pro-video/

https://twitter.com/ha…

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.me/papers Get Blender: https://www.blender.org/ Demo files: https://www.blender.org/download/demo-files/

Blend Swap: https://www.blendswap.com/ Andrew Price's donut tutorial:

https://www.youtube.com/watch?v=B0J27sf9N1Y&list=PLjEaoINr3zgEPv5y--4MKpciLaoQYZB1Z&index=2 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Be…

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 My Master thesis on fluids, with source code: https://users.cg.tuwien.ac.at/zsolnai/gfx/fluid_control_msc_thesis/

📝 Paper/poster on fluid control, with source code: https://users.cg.tuwien.ac.at/zsolnai/gfx/real_time_fluid_control_eg/ 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Bri…

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.me/papers The conference:

https://wandb.ai/site/resources/events/fully-connected First impressions from creatives: https://openai.com/blog/sora-first-impressions 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald…

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Putra Iskandar, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tyb…

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 The paper "A generalist AI agent for 3D virtual environments" is available here:

https://deepmind.google/discover/blog/sima-generalist-ai-agent-for-3d-virtual-environments/ 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Ky…

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambdalabs.com/papers 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Putra Iskandar, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tyb…

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.me/papers 📝 The paper "ComPromptMized: Unleashing Zero-click Worms that Target GenAI-Powered Applications" is available here:

https://sites.google.com/view/compromptmized 📝 My paper on simulations that look almost like reality is available for free here:

https://rdcu.be/cWPfD Or this is the orig. Nature Physics link with clickable citations:

https://www.nature.com/articles/s41567-022-01788-5 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Luka…

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.me/papers 📝 Claude 3 is available here - try it out for free (note that we are not affiliated with them):

https://www.anthropic.com/news/claude-3-family Conference I am coming to:

https://wandb.ai/site/resources/events/fully-connected Experiments, evaluations:

https://twitter.com/JiaweiLiu_/status/1764866009175920892

https://twitter.com/tolgabilge_/status/1764754012824314102

https://twitter.com/RubenHssd/status/1764692641436827842 Additional results (very nice so far!):

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 📝 My paper on simulations that look almost like reality is available for free here:

…

Подписывайтесь - https://dzen.ru/spbai !

Участники: - Петр Ермаков, ML Brand Director, Яндекс

- Иван Барабанов, Разработчик, ВКонтакте, deep vk

- Валентин Хрульков, ведущий исследователь, Yandex Research Денис Димитров, Исполнительный директор по исследованию данных Sber AI, научный консультант AIRI Вместе со специалистами в области диффузионных картиночных моделей порассуждаем, куда развивается область. Поговорим про текущие положение дел и актуальные технические барьеры области.

Спикер: Валентин Хрульков, ведущий исследователь, Yandex Research Рассмотрим все этапы обучения foundational диффузионных моделей, начиная от подготовки датасета до регулярных замеров качества в процессе обучения. Обсудим scaling law эксперименты и их предварительные результаты. Так же обсудим различные аспекты применения этих моделей на практике: генерации рекламных баннеров, персонализация, сервис-социальная сеть Шедеврум.

Спикер: Андрей Кузнецов, Исполнительный директор по исследованию данных, Sber AI Рассмотрим теорию и доступную информацию о диффузионном процессе. Подробно покажу, чем отличается архитектура Kandinsky 2.1 от 2.2. Обсудим метрики для оценки качества генераций, поговорим про ключевые результаты релизов. Вместе посмотрим, в каких сценариях для бизнеса и практических приложениях можно применять нашу модель.

Добро пожаловать на вечерний митап для ML-инженеров от Яндекса. В этот раз поговорим про модные нынче «генеративки», а именно про диффузионные картиночные модели! программа:

00:00:00 – таймер обратного отсчёта перед эфиром

00:02:12 – открытие конференции

00:04:52 – Kandinsky 2.X

Андрей Кузнецов, исполнительный директор по исследованию данных, Sber AI.

00:46:55 – перерыв

01:03:40 – Практические аспекты обучения масштабных диффузионных моделей Валентин Хрульков, ведущий исследователь, Yandex Research. 01:49:34 – Дискуссия «Ближайшее будущее диффузионных моделей» Присоединяйтесь к нашем сообществу в телеграм, чтобы быть в курсе всех событий и мероприятий Яндекса https://t.me/yadatadojo. Вопрос…

Спикеры: Дмитрий Колодезев, Алексей Пустынников, Алексей Натекин Страница соревнования: https://ods.ai/tracks/data-fusion-2024-competitions

Дедлайн по соревнованию 5 апреля 2024 года, присоединяйтесь!

Спикеры: Алексей Натекин, Дмитрий Колодезев Страница соревнования: https://ods.ai/tracks/data-fusion-2024-competitions

Все презентации можно скачать на странице митапа https://ods.ai/tracks/data-fusion-2024-competitions/meetup Дедлайн по соревнованию 5 апреля 2024 года, присоединяйтесь!

С радостью приглашаем вас на уникальное событие, посвященное силе и вкладу женщин в мире данных - митап "Woman In Data Science"! Это не просто встреча, это праздник ума, таланта и вдохновения, организованный ODS SPB при поддержке компании Samokat.tech. Полная программа доступна на ODS:

https://ods.ai/events/wids-meetup-2024 Вступить в сообщество: https://ods.ai/ Соцсети Data Fest & Course Fest: https://t.me/datafest

https://vk.com/datafest

Курс Open ML Course: Деревья и их ансамбли:

https://ods.ai/tracks/trees-autumn23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Open ML Course: Деревья и их ансамбли:

https://ods.ai/tracks/trees-autumn23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Deep Reinforcement Learning 2023: https://ods.ai/tracks/drlcourse23

Сезон курсов:https://ods.ai/events/course_season_autumn_23 В четвертой лекции:

- Рассматривается случай MDP с неизвестными функциями награды и перехода между состояниями

- Рассмотрели подход Monte-Carlo и Temporal-Difference для нахождения Q-функции в этом случае

- Обсудили epsilon-жадные политики

- Вывили алгоритмы Monte-Carlo, SARSA и Q-learning Автор курса: Антон Плаксин, исследователь в группе Yandex.Research и доцент Уральского федерального университета. Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам:…

Курс Deep Reinforcement Learning 2023: https://ods.ai/tracks/drlcourse23

Сезон курсов:https://ods.ai/events/course_season_autumn_23 В третьей лекции:

- Поговорили про принцип динамического программирования

- Рассмотрели понятия v- и q-функций, а также понятия оптимальной политики.

- Выписали уравнения Белламана и научились их решать методами Policy Iteration и Value Iteration. Автор курса: Антон Плаксин, исследователь в группе Yandex.Research и доцент Уральского федерального университета. Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат …

Курс Deep Reinforcement Learning 2023: https://ods.ai/tracks/drlcourse23

Сезон курсов:https://ods.ai/events/course_season_autumn_23 На втором практическом занятии: - Разбираемся с PyTorch

- Пишем линейную регрессию

- Решаем задачу регрессии с помощь нейронных сетей

- Реализуем Deep Cross-Entropy метод и решаем CartPole Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Open ML Course: Линейные модели 2023: https://ods.ai/tracks/linear-models-autumn23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

На третьем практическом занятии: - Разбираемся с со средой Frozen Lake

- Пишем Policy Iteration Автор курса: Антон Плаксин, исследователь в группе Yandex.Research и доцент Уральского федерального университета. Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Open ML Course: Линейные модели 2023: https://ods.ai/tracks/linear-models-autumn23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Страница курса: https://ods.ai/tracks/my_first_data_project

Все доп.материалы в блоке на странице курса: https://ods.ai/tracks/my_first_data_project/blocks/98c41cb4-aaff-4c5e-8ddf-252be36ed722 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Open ML Course: Деревья и их ансамбли:

https://ods.ai/tracks/trees-autumn23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс ML System Design: https://ods.ai/tracks/ml-system-design-23

Сезон курсов: https://ods.ai/events/course_season_autumn_23 Наши соц.сети:

Telegram: https://t.me/datafest

Вконтакте: https://vk.com/datafest

Канал с вакансиями в telegram: https://t.me/odsjobs

Канал с апдейтами по курсам: https://t.me/odscourses

Как попасть в чат сообщества ODS Mattermost: https://ods.ai/tracks/mattermost

Курс Deep Reinforcement Learning 2023: https://ods.ai/tracks/drlcourse23

Сезон курсов:https://ods.ai/events/course_season_autumn_23 Во второй лекции:

- рассмотрены понятия нейрона, функции активации, нейронных сетей;

- кратко изложен нейросетевой подход к решению задач регрессии и классификации;

- приведена Теорема Цибенко об аппроксимации нейронными сетями непрерывных функций;

- рассказана модификация алгоритма Кросс-Энтропии с использованием нейронных сетей для решения задач обучения с подкреплением с бесконечными пространствами состояний и действий. Автор курса: Антон Плаксин, исследователь в группе Yandex.Research и доцент Уральского федерального университета. Наши соц.сети:

Telegram: h…

Twitch: https://www.twitch.tv/primerjustin

Discord: https://discord.gg/NbruaNW

Store: https://store.dftba.com/collections/primer

Patreon: https://www.patreon.com/primerlearning

Twitter: @primerlearning 0:00 - Introduction

0:19 - Simulation rules

3:23 - First simulations

5:21 - Game theory analysis

8:45 - Alternate reward matrices

15:58 - Requirements for an evolutionarily stable strategy

16:69 - Discussion questions Made possible by support from these wonderful Patrons:

abledbody

Alba Caparros-Roissard

Andrew Lang

Anthony Eufemio

Brian Cloutier

Captain Chinchilla

Christy Serbus

Daniel Rolandsgard Kjellevold

Erik Broeders

Flavio Kindler

Gabriele Siino

Garrett

Guguke

James Manning

Jeff

Jeremy…

Sean Carroll is a theoretical physicist, author, and host of Mindscape podcast.

Please support this podcast by checking out our sponsors:– HiddenLayer: https://hiddenlayer.com/lex– Cloaked: https://cloaked.com/lex and use code LexPod to get 25% off– Notion: https://notion.com/lex– Shopify: https://shopify.com/lex to get $1 per month trial– NetSuite: http://netsuite.com/lex to get free product tourTranscript: https://lexfridman.com/sean-carroll-3-transcriptEPISODE LINKS:Sean’s Website: https://preposterousuniverse.comMindscape Podcast: https://www.preposterousuniverse.com/podcast/Sean’s YouTube: https://youtube.com/@seancarrollSean’s Patreon: https://www.patreon.com/seanmcarrollSean’s Twitte…

Neil Adams is a judo world champion, 2-time Olympic silver medalist, 5-time European champion, and often referred to as the Voice of Judo.

Please support this podcast by checking out our sponsors:– ZipRecruiter: https://ziprecruiter.com/lex– Eight Sleep: https://eightsleep.com/lex to get special savings– MasterClass: https://masterclass.com/lexpod to get 15% off– LMNT: https://drinkLMNT.com/lex to get free sample pack– NetSuite: http://netsuite.com/lex to get free product tourEPISODE LINKS:Neil’s Instagram: https://instagram.com/naefightingNeil’s YouTube: https://youtube.com/NAEffectiveFightingNeil’s TikTok: https://tiktok.com/@neiladamsmbeNeil’s Facebook: https://facebook.com/NeilAdamsJudo…

Edward Gibson is a psycholinguistics professor at MIT and heads the MIT Language Lab.

Please support this podcast by checking out our sponsors:– Yahoo Finance: https://yahoofinance.com– Listening: https://listening.com/lex and use code LEX to get one month free– Policygenius: https://policygenius.com/lex– Shopify: https://shopify.com/lex to get $1 per month trial– Eight Sleep: https://eightsleep.com/lex to get special savingsTranscript: https://lexfridman.com/edward-gibson-transcriptEPISODE LINKS:Edward’s X: https://x.com/LanguageMITTedLab: https://tedlab.mit.edu/Edward’s Google Scholar: https://scholar.google.com/citations?user=4FsWE64AAAAJTedLab’s YouTube: https://youtube.com/@Tedlab-MITP…

Andrew Callaghan is the host of Channel 5 on YouTube, where he does street interviews with fascinating humans at the edges of society, the so-called vagrants, vagabonds, runaways, outlaws, from QAnon adherents to Phish heads to O Block residents and much more.

Please support this podcast by checking out our sponsors:– ShipStation: https://shipstation.com/lex and use code LEX to get 60-day free trial– BetterHelp: https://betterhelp.com/lex to get 10% off– LMNT: https://drinkLMNT.com/lex to get free sample pack– MasterClass: https://masterclass.com/lexpod to get 15% off– AG1: https://drinkag1.com/lex to get 1 month supply of fish oilTranscript: https://lexfridman.com/andrew-callaghan-transcri…

Bassem Youssef is an Egyptian-American comedian & satirist, referred to as the Jon Stewart of the Arab World.

Please support this podcast by checking out our sponsors:– AG1: https://drinkag1.com/lex to get 1 month supply of fish oil– Shopify: https://shopify.com/lex to get $1 per month trial– Eight Sleep: https://eightsleep.com/lex to get special savings– LMNT: https://drinkLMNT.com/lex to get free sample packEPISODE LINKS:Bassem’s X: https://x.com/ByoussefBassem’s Instagram: https://instagram.com/bassemBassem’s Facebook: https://facebook.com/bassemyousseftvBassem’s Website: https://bassemyoussef.xyzPODCAST INFO:Podcast website: https://lexfridman.com/podcastApple Podcasts: https://apple.co…

Tulsi Gabbard is a politician, veteran, and author of For Love of Country.

Please support this podcast by checking out our sponsors:– Riverside: https://creators.riverside.fm/LEX and use code LEX to get 30% off– ExpressVPN: https://expressvpn.com/lexpod to get 3 months free– NetSuite: http://netsuite.com/lex to get free product tour– Notion: https://notion.com/lexEPISODE LINKS:For Love of Country (book): https://amzn.to/3VLlofMTulsi’s X: https://x.com/tulsigabbardTulsi’s YouTube: https://youtube.com/@TulsiGabbardTulsi’s Podcast: https://youtube.com/@TheTulsiGabbardShowTulsi’s Instagram: https://instagram.com/tulsigabbardTulsi’s Facebook: https://facebook.com/TulsiGabbardTulsi’s Website: htt…

Mark Cuban is a businessman, investor, star of TV series Shark Tank, long-time principal owner of Dallas Mavericks, and founder of Cost Plus Drugs.

Please support this podcast by checking out our sponsors:– Listening: https://listening.com/lex and use code LEX to get one month free– Cloaked: https://cloaked.com/lex and use code LexPod to get 25% off– Notion: https://notion.com/lex– Eight Sleep: https://eightsleep.com/lex to get special savings– Shopify: https://shopify.com/lex to get $1 per month trialEPISODE LINKS:Mark’s X: https://twitter.com/mcubanMark’s Instagram: https://instagram.com/mcubanCost Plus Drugs: https://costplusdrugs.comShark Tank: https://abc.com/shows/shark-tankDallas Mav…

Dana White is the CEO and president of the UFC.

Please support this podcast by checking out our sponsors:– LMNT: https://drinkLMNT.com/lex to get free sample pack– Notion: https://notion.com/lex– AG1: https://drinkag1.com/lex to get 1 month supply of fish oil– InsideTracker: https://insidetracker.com/lex to get 20% offTranscript: https://lexfridman.com/dana-white-transcriptEPISODE LINKS:Dana’s X: https://x.com/danawhiteDana’s Instagram: https://instagram.com/danawhiteDana’s Facebook: https://facebook.com/danawhiteUFC’s YouTube: https://youtube.com/@UFCUFC’s Website: https://ufc.com/PODCAST INFO:Podcast website: https://lexfridman.com/podcastApple Podcasts: https://apple.co/2lwqZIrSpotify: h…

Annie Jacobsen is an investigative journalist and author of “Nuclear War: A Scenario” and many other books on war, weapons, government secrecy, and national security.

Please support this podcast by checking out our sponsors:– HiddenLayer: https://hiddenlayer.com/lex– BetterHelp: https://betterhelp.com/lex to get 10% off– Policygenius: https://policygenius.com/lex– NetSuite: http://netsuite.com/lex to get free product tourEPISODE LINKS:Nuclear War: A Scenario (book): https://amzn.to/3THZHfrAnnie’s Twitter: https://twitter.com/anniejacobsenAnnie’s Website: https://anniejacobsen.com/Annie’s Books: https://amzn.to/3TGWyMJAnnie’s Books (audio): https://adbl.co/49ZnI7cPODCAST INFO:Podcast website…

Sam Altman is the CEO of OpenAI, the company behind GPT-4, ChatGPT, Sora, and many other state-of-the-art AI technologies.

Please support this podcast by checking out our sponsors:– Cloaked: https://cloaked.com/lex and use code LexPod to get 25% off– Shopify: https://shopify.com/lex to get $1 per month trial– BetterHelp: https://betterhelp.com/lex to get 10% off– ExpressVPN: https://expressvpn.com/lexpod to get 3 months freeTranscript: https://lexfridman.com/sam-altman-2-transcriptEPISODE LINKS:Sam’s X: https://x.com/samaSam’s Blog: https://blog.samaltman.com/OpenAI’s X: https://x.com/OpenAIOpenAI’s Website: https://openai.comChatGPT Website: https://chat.openai.com/Sora Website: https://op…

Norman Finkelstein and Benny Morris are historians.

Mouin Rabbani is a Middle East analyst.

Steven Bonnell (aka Destiny) is a political livestreamer.

On some podcast players you should be able to click the timestamp to jump to that time.

(00:00) – Introduction(12:11) – 1948(1:10:43) – Partition(2:15:16) – October 7(3:09:27) – Gaza(3:36:02) – Peace(4:40:47) – Hope for the future

Kimbal Musk is a chef, entrepreneur, and author of The Kitchen Cookbook: Cooking for Your Community.

Please support this podcast by checking out our sponsors:– Eight Sleep: https://eightsleep.com/lex to get special savings– ExpressVPN: https://expressvpn.com/lexpod to get 3 months free– NetSuite: http://netsuite.com/lex to get free product tour– BetterHelp: https://betterhelp.com/lex to get 10% offTranscript: https://lexfridman.com/kimbal-musk-transcriptEPISODE LINKS:Kimbal’s X: https://x.com/kimbalKimbal’s Instagram: https://instagram.com/kimbalmusk/Kimbal’s Facebook: https://facebook.com/kimbalmuskofficial/The Kitchen Cookbook: https://amzn.to/4ccaCoEThe Kitchen (restaurants): https://www…

Yann LeCun is the Chief AI Scientist at Meta, professor at NYU, Turing Award winner, and one of the most influential researchers in the history of AI.

Please support this podcast by checking out our sponsors:– HiddenLayer: https://hiddenlayer.com/lex– LMNT: https://drinkLMNT.com/lex to get free sample pack– Shopify: https://shopify.com/lex to get $1 per month trial– AG1: https://drinkag1.com/lex to get 1 month supply of fish oilEPISODE LINKS:Yann’s Twitter: https://twitter.com/ylecunYann’s Facebook: https://facebook.com/yann.lecunMeta AI: https://ai.meta.com/PODCAST INFO:Podcast website: https://lexfridman.com/podcastApple Podcasts: https://apple.co/2lwqZIrSpotify: https://spoti.fi/2nEwCF8R…

Serhii Plokhy is a Ukrainian historian at Harvard University, director of the Ukrainian Research Institute, and an author of many books on history of Eastern Europe, including his latest book The Russo-Ukrainian War: The Return of History.

Please support this podcast by checking out our sponsors:– Eight Sleep: https://eightsleep.com/lex to get special savings– Shopify: https://shopify.com/lex to get $1 per month trial– NetSuite: http://netsuite.com/lex to get free product tour– AG1: https://drinkag1.com/lex to get 1 month supply of fish oilEPISODE LINKS:Serhii’s X: https://x.com/splokhySerhii’s Website: https://history.fas.harvard.edu/people/serhii-plokhiiHarvard Ukrainian Research Institut…

Tucker Carlson is a highly-influential political commentator.

You can watch and listen to him on the Tucker Carlson Network and the Tucker Carlson Podcast.

Please support this podcast by checking out our sponsors:– ZipRecruiter: https://ziprecruiter.com/lex– Listening: https://listening.com/lex and use code LEX to get one month free– HiddenLayer: https://hiddenlayer.com/lex– LMNT: https://drinkLMNT.com/lex to get free sample pack– AG1: https://drinkag1.com/lex to get 1 month supply of fish oilTranscript: https://lexfridman.com/tucker-carlson-transcriptEPISODE LINKS:Tucker Carlson Network: https://tuckercarlson.com/Tucker Carlson Podcast: https://tuckercarlson.com/listen/Tucker’s X: https://…

Well, I’ve heard other people on your teams use words like surprise, sometimes even shock …HOSN: Yeah, yeah, there are a lot of “wow” factors.

HUIZINGA: Yeah, yeah.

AI research is moving at such speeds that I feel like we need to get accustomed to a timing of now.

HOSN: That’s right.

Well, as we close, Rafah, I want to ask a question anchored on the big idea behind AI Frontiers.

GRETCHEN HUIZINGA: Welcome to Abstracts, a Microsoft Research Podcast that puts the spotlight on world-class research in brief.

CHAKRABORTY: So satellite connectivity is nothing new and has been there for long.

So we are talking about the satellites that are at least 10 to 20 times cheaper and smaller than state-of-the-art satellites.

So the device sends some packet to the satellite; satellite sends some packet to the device—it’s all about packet exchange.

So our vision is clear: to bring 24-7 connectivity for devices anywhere on Earth with just a click of power button.

Behind every emerging technology is a great idea propelling it forward. In the new Microsoft Research Podcast series, Ideas, members of the research community at Microsoft discuss the beliefs that animate their research, the experiences and thinkers that inform it, and the positive human impact it targets. In this episode, host Gretchen Huizinga talks with Principal Researcher Kalika Bali. Inspired by an early vision of “talking computers” and a subsequent career in linguistics, Bali has spent the last two decades bringing the two together. Aided by recent advances in large language models and motivated by her belief that everyone should have access to AI in their own language, Bali and her…

And so I just want to start here: for you, Ida, what is general intelligence?

Different people at different times provide different criteria for what would be the artificial general intelligence notion.

One is artificial general intelligence and the other is humanlike intelligence or human-level intelligence.

Artificial general intelligence and humanlike, human-level intelligence—how do these two concepts relate to you?

LLORENS: So it sounds like a very extensive set of experiments across many different tasks and with many different leading AI models, and you’ve uncovered a lack of robustness across some of these different tasks.

GRETCHEN HUIZINGA: Welcome to Abstracts, a Microsoft Research Podcast that puts the spotlight on world-class research in brief.

These two examples are also the differences from other deep learning OFDFT works.

This is the generalization challenge and is one of the major challenges of deep learning method for molecular science applications.

This somehow shows the benefits of leveraging the OFDFT framework for using a deep learning method to solve molecular tasks.

You can also read it on arXiv, or you can check out the March 2024 issue of Nature Computational Science.

And so we realized that working with generative AI really parallels these different aspects of what a manager does, right.

So this requires having self-awareness of the applicability of generative AI to your workflows and maintaining an appropriate level of confidence in completing tasks manually or relying on generative AI.

For example, whether it’s worth it for you to actually learn how to work with generative AI more effectively.

But I think, given how generative AI has rolled out in the world today, I mean, a lot of the focus has been on productivity and workflows.

If you want to read the full paper on metacognition and generative AI, you can find a link at aka.ms/abstracts, or you can …

NICOLE FORSGREN: Yeah, it’s, you know, it’s, kind of, this ridiculous story.

I was there for, you know, two, three years, and I’m doing really, really well.

GEHRKE: This is just “I had a feeling.”FORSGREN: In my gut, I’m like, I’m doing really well.

After that, things were going really well, but we were also growing and scaling really, really rapidly.

Because I realized there were pieces about research that I really, really loved.

In the Microsoft Research Podcast series What’s Your Story, Johannes Gehrke explores the who behind the technical and scientific advancements helping to reshape the world.

A systems expert whose 10 years with Microsoft spans research and product, Gehrke talks to members of the company’s research community about what motivates their work and how they got where they are today.

Partner Software Architect Ivan Tashev’s expertise in audio signal processing has contributed to the design and study of audio components for Microsoft products such as Kinect, Teams, and HoloLens.

In this episode, Tashev discusses how a first-place finish in the Mathematical Olympiad fueled a lifelong passion for shoot…

And up until now, there’s been a lot of work on pruning model parameters for a variety of reasons.

But generally, these papers show that as parameters are removed from the model, performance just does not degrade.

You can, overall, keep performance roughly the same even with a fairly drastic reduction of model parameters.

HUIZINGA: So, Jordan, I often think of an abstract as a sort of appetizer for a research paper.

I think for one, as a practical matter, there’s this question of just what’s the best way to find the best LASER intervention?

LLORENS: Your new book lays out foundations in statistics and probability theory for modern machine learning.

LLORENS: Another concept that is key in machine learning is generalization.

So that’s, that’s … we show that in the book, in fact.

BISHOP: [LAUGHS] Well, that’s, that’s really interesting.

So I personally, actually, find this one of the most exciting frontiers not only of the natural sciences but also of machine learning.

Members of the research community at Microsoft work continuously to advance their respective fields.

Abstracts brings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.

In this episode, Senior Principal Research Manager Tao Qin and Senior Researcher Lijun Wu discuss “FABind: Fast and Accurate Protein-Ligand Binding.” The paper, accepted at the 2023 Conference on Neural Information Processing Systems (NeurIPS), introduces a new method for predicting the binding structures of proteins and ligands during drug development.

The method demonstrates improved speed and accuracy over current methods.

Learn moreFABind: Fast and Ac…

Members of the research community at Microsoft work continuously to advance their respective fields.

Abstracts brings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.

In this episode, Principal Researcher Alessandro Sordoni joins host Gretchen Huizinga to discuss “Joint Prompt Optimization of Stacked LLMs using Variational Inference.” In the paper, which was accepted at the 2023 Conference on Neural Information Processing Systems (NeurIPS), Sordoni and his coauthors introduce Deep Language Networks, or DLNs, an architecture that treats large language models as layers within a network and natural language prompts as eac…

Members of the research community at Microsoft work continuously to advance their respective fields.

Abstracts brings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.

In this episode, Xing Xie, a Senior Principal Research Manager of Microsoft Research Asia, joins host Dr. Gretchen Huizinga to discuss “Evaluating General-Purpose AI with Psychometrics.” As AI capabilities move from task specific to more general purpose, the paper explores psychometrics, a subfield of psychology, as an alternative to traditional methods for evaluating model performance and for supporting consistent and reliable systems.

Read the paper: Ev…

Members of the research community at Microsoft work continuously to advance their respective fields.

Abstracts brings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.

In this episode, Xing Xie, a Senior Principal Research Manager of Microsoft Research Asia, joins host Dr. Gretchen Huizinga to discuss “Evaluating General-Purpose AI with Psychometrics.” As AI capabilities move from task specific to more general purpose, the paper explores psychometrics, a subfield of psychology, as an alternative to traditional methods for evaluating model performance and for supporting consistent and reliable systems.

Read the paper: Ev…

It often requires the knowledge and experience of individuals from across disciplines and institutions.

Collaborators, a Microsoft Research Podcast series, explores the relationships—both expected and unexpected—behind the projects, products, and services being pursued and delivered by researchers at Microsoft and the diverse range of people they’re teaming up with.

In this episode, Dr. Gretchen Huizinga speaks with Cecily Morrison, MBE, a Senior Principal Research Manager at Microsoft Research, and Karolina Pakėnaitė, who also goes by Caroline, a PhD student and member of the citizen design team working with Morrison on the research project Find My Things.

An AI phone application designed …

Curious about the safety of LLMs?

🤔 Join us for an insightful new episode featuring Suchin Gururangan, Young Investigator at Allen Institute for Artificial Intelligence and Data Science Engineer at Appuri.

🚀 Don't miss…

This podcast episode features Dr. Mohamed Elhoseiny, a true luminary in the realm of computer vision with over a decade of groundbreaking research.

As an Assistant Professor at KAUST, Dr. Elhoseiny's work delves into the…

Our first guest with this new format is Kyle Lo, the most senior lead scientist in the Semantic Scholar team at Allen Institute for AI (AI2), who kindly agreed to share his perspective on #Science of #Science (#scisci) o…

She is interested in studying and understanding the neural mechanism of the honeybee waggle dance.

Barbara and Anna shared some breakthroughs in the field of animal communication.

Anna discussed how the honeybee uses the waggle dance to communicate.

Our guests explained how they captured the waggle dance of honeybees in their hives.

She also discussed how researchers use the neural socket to understand the workings of insects' brains.

Talmo PereiraDr. Talmo Pereira is a Principal Investigator at the Salk Institute for Biological Studies in San Diego, CA where he leads a research group as a Salk Fellow.

His lab (talmolab.org) focuses on the development of deep learning-based computational methods for recognition and modeling of complex biological systems, with applications ranging from neuroscience to cancer and plant biology.

His recent work has demonstrated how advances in deep learning and computer vision can enable quantitative phenotyping of complex behaviors through the development and application of approaches for markerless motion capture (sleap.ai).

This work has been published in Nature Methods and featured in T…

Modeling Group BehaviorOur guest in this episode is Sebastien Motsch, an assistant professor at Arizona State University, working in the School of Mathematical and Statistical Science.

Sebastien discussed two approaches to modeling the group behavior of animals, for instance, a flock of birds.

He discussed how Boltzmann's questions and kinetic theory help them understand birds' interactions with respect to velocity changes.

Papers discussedA new model for self-organized dynamics and its flocking behaviorHeterophilious dynamics enhances consensusResourceBoltzmann equation$$f(t + \Delta t, \mathbf{x} + \Delta \mathbf{x}, \mathbf{v} + \Delta \mathbf{v}) = f(t, \mathbf{x}, \mathbf{v}) + \left( …

Ryan discussed how the behavior of rattlesnakes is studied in the natural world, particularly with an increase in temperature.

He discussed how they collect data about the rattlesnake hunt using loggers and how they determine the loggers do not affect the animals' behavior.

Ryan discussed how he built a machine learning model to predict the behavior of the animals.

Ryan discussed what he discovered about the eating habits of rattlesnakes.

Rounding up, Ryan shared some future plans for the project.

What you Know About Intelligence is WrongWe are joined by Hank Schlinger, a professor of psychology at California State University, Los Angeles.

His research revolves around theoretical issues in psychology and behavioral analysis.

He also discussed how intelligence is measured in a given context.

Hank mentioned why the current measure of intelligence is fundamentally flawed.

Hank discussed how psychologists can perform behavioral experiments to understand consciousness.

What you Know About Intelligence is WrongWe are joined by Hank Schlinger, a professor of psychology at California State University, Los Angeles.

His research revolves around theoretical issues in psychology and behavioral analysis.

He also discussed how intelligence is measured in a given context.

Hank mentioned why the current measure of intelligence is fundamentally flawed.

Hank discussed how psychologists can perform behavioral experiments to understand consciousness.

Louis and the interim director at the Whitney R. Harris World Ecology Center.

Aimee discussed how animals perceive information and what they use it for.

She also discussed the costs required for learning and factors that affect animal learning.

She also discussed the different kinds of evolutionary experiments that can be performed.

Aimee discussed some models that researchers use during evolutionary experiments.

Octopus CognitionWe are joined by Tamar Gutnick, a visiting professor at the University of Naples Federico II, Napoli, Italy.

She studies the octopus nervous system and their behavior, focusing on cognition and learning behaviors.

Tamar gave a background to the kind of research she does — lab research.

She discussed the octopus nervous system and why they are unique compared to other animals.

She discussed how they measure the behavior of octopuses using a video recording and a logger to track brain activity.

Claire discussed how bumblebees make foraging decisions and how they communicate when foraging.

She discussed how they set up experiments in the lab to address questions about bumblebees foraging.

Claire discussed factors that drive an animal's foraging decisions.

She also touched on some irrational foraging behaviors she observed in her study.

She discussed the effect of climate change on foraging bees' learning behavior.

On today’s show, we are joined by our co-host, Becky Hansis-O’Neil. Becky is a Ph.D. student at the University of Missouri, St Louis, where she studies bumblebees and tarantulas to understand their learning and cognitive work. She joins us to discuss the paper: Perception in Chess. The paper aimed to understand how chess players perceive the positions of chess pieces on a chess board. She discussed the findings paper. She spoke about situations where grandmasters had better recall of chess positions than beginners and situations where they did not. Becky and Kyle discussed the use of chess engines for cheating. They also discussed how chess players use chunking. Becky discussed some approac…

OpenwormOn this episode, we are joined by Stephen Larson, the CEO of MetaCell and an affiliate of the OpenWorm foundation.

Stephen discussed what the Openworm project is about.

They hope to use a digital C. elegans nematode (C. elegans for short) to study the basics of life.

Stephen discussed why C. elegans is an ideal organism for studying life in the lab.

He also mentioned how students can get involved in the Openworm project.

What the Antlion KnowsOur guest is Becky Hansis-O’Neil, a Ph.D. student at the University of Missouri, St Louis.

Becky discussed how they designed an experiment using a T-maze to understand antlions' behavior.

Becky discussed some interesting findings from the experiment.

Becky gave her thoughts on the findings of the paper and the methodologies used.

Paper in focusOperant conditioning in antlion larvae and its impairment following exposure to elevated temperaturesFollow our guestXWebsiteThis season’s cover art features original photographs by Becky Hansis-O’Neil

Kyle is joined by friends and former guests Pramit Choudhary and Frank Bell to have an open discussion of the impacts LLMs and machine learning have had in the past year on industry, and where things may go in the current year.

Uncontrollable AI RisksWe are joined by Darren McKee, a Policy Advisor and the host of Reality Check — a critical thinking podcast.

Darren gave a background about himself and how he got into the AI space.

Darren shared his thoughts on AGI's achievements in the coming years.

Darren discussed his worry about AI surpassing human understanding of the universe and potentially causing harm to humanity.

He explored whether AI possesses inherently evil intentions and gave his thoughts on regulating AI.

Tidyverse, ggplot2, and the secret to a tech company’s longevity: Hadley Wickham talks to Jon Krohn about Posit’s rebrand, Tidyverse and why it needs to be in every data scientist’s toolkit, and why getting your hands di…

Mixtral 8x22B is the focus on this week's Five-Minute Friday.

Jon Krohn examines how this model from French AI startup Mistral leverages its mixture-of-experts architecture to redefine efficiency and specialization in AI…

Generative AI is reshaping our world, and Bernard Marr, world-renowned futurist and best-selling author, joins Jon Krohn to guide us through this transformation.

In this episode, Bernard shares his insights on how AI is …

What are the risks of AI progressing beyond a point of no return?

What do we stand to gain?

On this Five-Minute Friday, Jon Krohn talks ‘books’ as he outlines two nonfiction works on AI and futurism by Oxford philosopher…

Tech entrepreneurship, artificial superintelligence, and the future of education: Aleksa Gordić speaks to Jon Krohn about his strategies for self-directed learning, the traits that help people succeed in moving from big …

Covariant's RFM-1: Jon Krohn explores the future of AI-driven robotics with RFM-1, a groundbreaking robot arm designed by Covariant and discussed by A.I.

roboticist Pieter Abbeel.

Explore how this innovation aims to merg…

Dr. Barrett Thomas, an award-winning Research Professor at the University of Iowa, explores the intricacies of Markov decision processes and their connection to Deep Reinforcement Learning.

Discover how these concepts ar…

Pytorch benefits, how to get funding for your AI startup, and managing scientific silos: In our new series for SuperDataScience, “In Case You Missed It”, host Jon Krohn engages in some “reinforcement learning through hum…

Kirill Eremenko joins Jon Krohn for another exclusive, in-depth teaser for a new course just released on the SuperDataScience platform, “Machine Learning Level 2”.

Kirill walks listeners through why decision trees and ra…

Explore the science of confidence with Lucy Antrobus, as she unveils neuroscience-backed strategies to build and boost confidence through practice, positive energy, and the power of laughter.

An essential listen for fost…

Generative AI in medicine takes center stage as Prof. Zachary Lipton, Chief Scientific Officer at Abridge, joins host Jon Krohn to discuss the significant advancements in AI that are reshaping healthcare.

This episode i…

Claude 3, LLMs and testing ML performance: Jon Krohn tests out Anthropic’s new model family, Claude 3, which includes the Haiku, Sonnet and Opus models (written in order of their performance power, from least to greatest…

Jon Krohn sits down with Sebastian Raschka to discuss his latest book, Machine Learning Q and AI, the open-source libraries developed by Lightning AI, how to exploit the greatest opportunities for LLM development, and wh…

Kurt Vonnegut's "Player Piano" delivers striking parallels between its dystopian vision and today's AI challenges.

This week, Jon Krohn explores the novel's depiction of a world where humans are marginalized by machines,…

In this episode of “Rust in the Cosmos” we delve into the challenge of testing software for… ehm … spaceHow can Rust help?

Build robotics applications in minutes, not months.

Amethix works to create and maximize the impact of the world’s leading corporations and startups, so they can create a better future for everyone they serve.

We provide solutions in AI/ML, Fintech, Defense, Robotics and Predictive maintenance.

CommunitiesAeroRust, Intrepid, BytenookAeroRust Discord invite: https://discord.com/invite/6jJyx5nEUqAeroRust website: AeroRust.orgIntrepid AI Discord https://discord.gg/cSSzche6Cthttps://discord.gg/cSSzche6Ct Intrepid AI website: https://intrepid.aiReferences

In this inaugural episode of “Rust in the Cosmos,” we delve into the intricacies of communication in space and some of the challenges in space application development.

In this episode we delve into the dynamic realm of game development and the transformative role of artificial intelligence (AI).

Join Frag, Jim and Mike as they explore the current landscape of game development processes, from initial creative ideation to the integration of AI-driven solutions.

With Mike’s expertise as a software executive and avid game developer, we uncover the potential of AI to revolutionize game design, streamline development cycles, and enhance player experiences.

SponsorsIntrepid AI is an AI assisted all-in-one platform for robotics teams.

Build robotics applications in minutes, not months.

In this episode, join me and the Kaggle Grand Master, Konrad Banachewicz, for a hilarious journey into the zany world of data science trends.

From algorithm acrobatics to AI, creativity, Hollywood movies, and music, we just can’t get enough.

It’s the typical episode with a dose of nerdy comedy you didn’t know you needed.

Buckle up, it’s a data disco, and we’re breaking down the binary!

SponsorsIntrepid AI is an AI assisted all-in-one platform for robotics teams.

In this episode of Data Science at Home, we explore the game-changing impact of low-code solutions in robotics development.

Discover how these tools bridge the coding gap, simplify integration, and enable trial-and-error development.

We’ll also uncover challenges with traditional coding methods using ROS.

Join us for a concise yet insightful discussion on the future of robotics!

Join us in a dynamic conversation with Yori Lavi, Field CTO at SQream, as we unravel the data analytics landscape.

From debunking the data lakehouse hype to SQream’s GPU-based magic, discover how extreme data challenges are met with agility.

Yori shares success stories, insights into SQream’s petabyte-scale capabilities, and a roadmap to breaking down organizational bottlenecks in data science.

Dive into the future of data analytics with SQream’s commitment to innovation, leaving legacy formats behind and leading the charge in large-scale, cost-effective data projects.

Tune in for a dose of GPU-powered revolution!

In this episode from a month ago, join me as we unravel the controversial CEO firing at OpenAI in December 2023.

I share my insights on the events, decode the intricacies, and explore what lies ahead for this influential organization.

Don’t miss this concise yet insightful take on the intersection of leadership and artificial intelligence innovation.

SponsorLearn what the new year holds for ransomware as a service, Active Directory, artificial intelligence and more when you download the 2024 Arctic Wolf Labs Predictions Report today at arcticwolf.com/datascience

!!WARNING!!

Due to some technical issues the volume is not always constant during the show.

I sincerely apologise for any inconvenienceFrancescoIn this episode, I speak with Richie Cotton, Data Evangelist at DataCamp, as he delves into the dynamic intersection of AI and education.

Richie, a seasoned expert in data science and the host of the podcast, brings together a wealth of knowledge and experience to explore the evolving landscape of AI careers, the skills essential for generative AI technologies, and the symbiosis of domain expertise and technical skills in the industry.

Dive into the world of Data Science at Home with our latest episode, where we explore the dynamic relationship between Artificial Intelligence and the redemption of open source software.

In this thought-provoking discussion, I share my insights on why now, more than ever, is the opportune moment for open source to leave an indelible mark on the field of AI.

Join me as I unpack my opinions and set expectations for the near future, discussing the pivotal role open source is set to play in shaping the landscape of data science and artificial intelligence.

Don’t miss out—tune in to gain a deeper understanding of this revolutionary intersection!

This episode is available as YouTube stream at htt…

In this captivating podcast episode, join renowned financial expert Chris Skinner as he delves into the fascinating realm of the future of money.

From cryptocurrencies to government currencies, the metaverse to artificial intelligence (AI), Skinner explores the intricate interplay between technology and humanity.

Gain valuable insights as he defines the future of money, examines the potential impact of cryptocurrencies on traditional government currencies, and addresses the advantages and disadvantages of digital currencies.

Brace yourself for an enlightening discussion on the integration of AI in the financial sector and its potential impact on humanity.

Once subscribed, you get full acces…

In this thought-provoking episode, we sit down with the renowned AI expert, Filip Piekniewski, Phd, who fearlessly challenges the prevailing narratives surrounding artificial general intelligence (AGI) and the singularity.

With a no-nonsense approach and a deep understanding of the field, Filip dismantles the hype and exposes some of the misconceptions about AI, LLMs and AGI.

If you’re seeking a refreshingly pragmatic perspective on the future of AI, this episode is an absolute must-listen.

Filip Piekniewski BioFilip Piekniewski is a distinguished computer vision researcher and engineer, specializing in visual object tracking and perception.

He is known for his realistic perspective on AI, …

Brace yourselves, dear friends!

In this episode, we delve into the earth-shattering revelation that OpenAI might have stumbled upon AGI (lol) and we’re all just seconds away from being replaced by highly sophisticated toasters (lol lol).

Spoiler alert: OpenAI’s CEO is just playing 7D chess with the entire human race.

So, sit back, relax, and enjoy this totally not ominous exploration into the ‘totally not happening’ future of AI!

Dive into the cool world of AI chips with us!

🚀 We’re breaking down how these special computer chips for AI have evolved and what makes them different.

Think of them like the superheroes of the tech world!

Don’t miss out!

🎙️🔍 #AIChips #TechTalk #SimpleScience

Hey there, engineering enthusiasts!

Ever wondered how engineers deal with the wild, unpredictable twists and turns in their projects?

In this episode, we’re spilling the beans on uncertainty and why it’s the secret sauce in every engineering recipe, not just the fancy stuff like deep learning and neural networks!

Join us for a ride through the world of uncertainty quantification.

Tune in and let’s demystify the unpredictable together!

In this episode, dive deep into the world of Language Models as we decode their intricate structure, revealing how these powerful algorithms exploit concepts from the past.

But… what if LLMs were just a database?

Referenceshttps://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering